2023-12-26T21:39:53.936 INFO:root:teuthology version: 0.0.1.dev226+g9680d2a 2023-12-26T21:39:53.936 DEBUG:teuthology.run:Teuthology command: teuthology --description fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --verbose --name yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828 --owner scheduled_yuriw@teuthology -- /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/orig.config.yaml 2023-12-26T21:39:53.987 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-26T21:39:54.079 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828 branch: wip-yuri-testing-2023-12-26-0957-quincy description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7501828' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific ' name: yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri-testing-2023-12-26-0957-quincy ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - \(POOL_APP_NOT_ENABLED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri-testing-2023-12-26-0957-quincy sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6259 sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 sleep_before_teardown: 0 subset: 111/12000 suite: fs suite_branch: wip-yuri-testing-2023-12-26-0957-quincy suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 targets: smithi059.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGAaTChfsyvSRP//8gkxCjUiJUT0U8hCULLVuGEMXUhbKRjMcXGWXuO95EFEdreiWwZxip00+jDqu5vPRYrcSA= smithi076.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJyxUj0593trmMuHzbKoaJL4pP7z9BfSI3SXn3ElkIWWNXbeqeVa+nPHdl4PRAe1dazFO/n+gfMpmPJeQcJ/4Xo= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - install: branch: pacific exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: pacific cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: docker.io/ceph/daemon-base:latest-pacific roleless: true - print: '**** done end installing pacific cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 9680d2aba1e76f64ade295c81599c97907457848 timestamp: 2023-12-26_19:48:51 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true ; do ceph orch ps ; ceph versions ; ceph fs dump; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1485078 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-12-26T21:39:54.079 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa; will attempt to use it 2023-12-26T21:39:54.080 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks 2023-12-26T21:39:54.080 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-12-26T21:39:54.082 INFO:teuthology.task.internal:Checking packages... 2023-12-26T21:39:54.106 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '5b5a60d2d510bfaa9213c41e32227fb008245a30' 2023-12-26T21:39:54.107 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:39:54.107 INFO:teuthology.packaging:ref: None 2023-12-26T21:39:54.107 INFO:teuthology.packaging:tag: None 2023-12-26T21:39:54.107 INFO:teuthology.packaging:branch: wip-yuri-testing-2023-12-26-0957-quincy 2023-12-26T21:39:54.107 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:39:54.107 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri-testing-2023-12-26-0957-quincy 2023-12-26T21:39:54.246 INFO:teuthology.task.internal:Found packages for ceph version 17.2.7-201.g5b5a60d2 2023-12-26T21:39:54.247 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-12-26T21:39:54.253 INFO:teuthology.task.internal:no buildpackages task found 2023-12-26T21:39:54.253 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-12-26T21:39:54.284 INFO:teuthology.task.internal:Saving configuration 2023-12-26T21:39:54.305 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-12-26T21:39:54.312 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-26T21:39:54.337 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-26 21:34:28.143534', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGAaTChfsyvSRP//8gkxCjUiJUT0U8hCULLVuGEMXUhbKRjMcXGWXuO95EFEdreiWwZxip00+jDqu5vPRYrcSA='} 2023-12-26T21:39:54.358 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-26 21:34:28.140704', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJyxUj0593trmMuHzbKoaJL4pP7z9BfSI3SXn3ElkIWWNXbeqeVa+nPHdl4PRAe1dazFO/n+gfMpmPJeQcJ/4Xo='} 2023-12-26T21:39:54.358 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-12-26T21:39:54.364 INFO:teuthology.task.internal:roles: ubuntu@smithi059.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-12-26T21:39:54.364 INFO:teuthology.task.internal:roles: ubuntu@smithi076.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-12-26T21:39:54.364 INFO:teuthology.run_tasks:Running task console_log... 2023-12-26T21:39:54.460 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fa13d57ae50>, signals=[15]) 2023-12-26T21:39:54.460 INFO:teuthology.run_tasks:Running task internal.connect... 2023-12-26T21:39:54.466 INFO:teuthology.task.internal:Opening connections... 2023-12-26T21:39:54.466 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi059.front.sepia.ceph.com 2023-12-26T21:39:54.468 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:39:54.537 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi076.front.sepia.ceph.com 2023-12-26T21:39:54.538 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:39:54.604 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-12-26T21:39:54.612 DEBUG:teuthology.orchestra.run.smithi059:> uname -m 2023-12-26T21:39:54.636 INFO:teuthology.orchestra.run.smithi059.stdout:x86_64 2023-12-26T21:39:54.636 DEBUG:teuthology.orchestra.run.smithi059:> cat /etc/os-release 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:NAME="CentOS Stream" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION="8" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:ID="centos" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:ID_LIKE="rhel fedora" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:VERSION_ID="8" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:PLATFORM_ID="platform:el8" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:ANSI_COLOR="0;31" 2023-12-26T21:39:54.698 INFO:teuthology.orchestra.run.smithi059.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-26T21:39:54.699 INFO:teuthology.orchestra.run.smithi059.stdout:HOME_URL="https://centos.org/" 2023-12-26T21:39:54.699 INFO:teuthology.orchestra.run.smithi059.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-26T21:39:54.699 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-26T21:39:54.699 INFO:teuthology.orchestra.run.smithi059.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-26T21:39:54.699 INFO:teuthology.lock.ops:Updating smithi059.front.sepia.ceph.com on lock server 2023-12-26T21:39:54.724 DEBUG:teuthology.orchestra.run.smithi076:> uname -m 2023-12-26T21:39:54.745 INFO:teuthology.orchestra.run.smithi076.stdout:x86_64 2023-12-26T21:39:54.746 DEBUG:teuthology.orchestra.run.smithi076:> cat /etc/os-release 2023-12-26T21:39:54.807 INFO:teuthology.orchestra.run.smithi076.stdout:NAME="CentOS Stream" 2023-12-26T21:39:54.807 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION="8" 2023-12-26T21:39:54.807 INFO:teuthology.orchestra.run.smithi076.stdout:ID="centos" 2023-12-26T21:39:54.807 INFO:teuthology.orchestra.run.smithi076.stdout:ID_LIKE="rhel fedora" 2023-12-26T21:39:54.807 INFO:teuthology.orchestra.run.smithi076.stdout:VERSION_ID="8" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:PLATFORM_ID="platform:el8" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:PRETTY_NAME="CentOS Stream 8" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:ANSI_COLOR="0;31" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:HOME_URL="https://centos.org/" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-12-26T21:39:54.808 INFO:teuthology.orchestra.run.smithi076.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-12-26T21:39:54.808 INFO:teuthology.lock.ops:Updating smithi076.front.sepia.ceph.com on lock server 2023-12-26T21:39:54.832 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-12-26T21:39:54.841 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-12-26T21:39:54.860 INFO:teuthology.task.internal:Checking for old test directory... 2023-12-26T21:39:54.860 DEBUG:teuthology.orchestra.run.smithi059:> test '!' -e /home/ubuntu/cephtest 2023-12-26T21:39:54.864 DEBUG:teuthology.orchestra.run.smithi076:> test '!' -e /home/ubuntu/cephtest 2023-12-26T21:39:54.886 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-12-26T21:39:54.892 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-12-26T21:39:54.893 DEBUG:teuthology.orchestra.run.smithi059:> test -z $(ls -A /var/lib/ceph) 2023-12-26T21:39:54.928 DEBUG:teuthology.orchestra.run.smithi076:> test -z $(ls -A /var/lib/ceph) 2023-12-26T21:39:54.952 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-12-26T21:39:55.002 INFO:teuthology.run_tasks:Running task kernel... 2023-12-26T21:39:55.016 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-12-26T21:39:55.017 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-12-26T21:39:55.017 DEBUG:teuthology.orchestra.run.smithi059:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-26T21:39:55.017 DEBUG:teuthology.orchestra.run.smithi076:> test -f /run/.containerenv -o -f /.dockerenv 2023-12-26T21:39:55.038 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T21:39:55.039 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2023-12-26T21:39:55.040 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T21:39:55.041 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2023-12-26T21:39:55.101 INFO:teuthology.orchestra.run.smithi076.stdout:4.18.0-489.el8.x86_64 2023-12-26T21:39:55.101 INFO:teuthology.task.kernel:Running kernel on smithi076: 4.18.0-489.el8.x86_64 2023-12-26T21:39:55.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum install -y kernel 2023-12-26T21:39:55.103 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-489.el8.x86_64 2023-12-26T21:39:55.104 INFO:teuthology.task.kernel:Running kernel on smithi059: 4.18.0-489.el8.x86_64 2023-12-26T21:39:55.104 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum install -y kernel 2023-12-26T21:39:59.049 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - AppStream 24 MB/s | 35 MB 00:01 2023-12-26T21:39:59.082 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - AppStream 26 MB/s | 35 MB 00:01 2023-12-26T21:40:07.749 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - BaseOS 26 MB/s | 57 MB 00:02 2023-12-26T21:40:07.815 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - BaseOS 28 MB/s | 57 MB 00:02 2023-12-26T21:40:16.058 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - Extras 70 kB/s | 18 kB 00:00 2023-12-26T21:40:16.575 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - Extras common packages 21 kB/s | 6.9 kB 00:00 2023-12-26T21:40:16.640 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras 75 kB/s | 18 kB 00:00 2023-12-26T21:40:17.085 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for python3-asyncssh owned by ceph 9.9 kB/s | 3.5 kB 00:00 2023-12-26T21:40:17.133 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras common packages 23 kB/s | 6.9 kB 00:00 2023-12-26T21:40:17.526 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 15 kB/s | 3.5 kB 00:00 2023-12-26T21:40:18.338 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 15 MB/s | 16 MB 00:01 2023-12-26T21:40:19.387 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 9.4 MB/s | 16 MB 00:01 2023-12-26T21:40:22.227 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 383 kB/s | 24 kB 00:00 2023-12-26T21:40:23.451 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 570 kB/s | 24 kB 00:00 2023-12-26T21:40:25.944 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-26T21:40:25.944 INFO:teuthology.orchestra.run.smithi076.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-26T21:40:26.237 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout:Install 3 Packages 2023-12-26T21:40:26.238 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:40:26.239 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 89 M 2023-12-26T21:40:26.239 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 96 M 2023-12-26T21:40:26.239 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:40:27.384 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-12-26T21:40:27.384 INFO:teuthology.orchestra.run.smithi059.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-12-26T21:40:27.580 INFO:teuthology.orchestra.run.smithi076.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 13 MB/s | 10 MB 00:00 2023-12-26T21:40:27.707 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi059.stdout: kernel x86_64 4.18.0-529.el8 baseos 10 M 2023-12-26T21:40:27.708 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-core x86_64 4.18.0-529.el8 baseos 43 M 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules x86_64 4.18.0-529.el8 baseos 36 M 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout:Install 3 Packages 2023-12-26T21:40:27.709 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 89 M 2023-12-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 96 M 2023-12-26T21:40:27.710 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:40:28.446 INFO:teuthology.orchestra.run.smithi076.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2023-12-26T21:40:28.861 INFO:teuthology.orchestra.run.smithi059.stdout:(1/3): kernel-4.18.0-529.el8.x86_64.rpm 14 MB/s | 10 MB 00:00 2023-12-26T21:40:28.980 INFO:teuthology.orchestra.run.smithi076.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 20 MB/s | 43 MB 00:02 2023-12-26T21:40:28.980 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:40:28.980 INFO:teuthology.orchestra.run.smithi076.stdout:Total 32 MB/s | 89 MB 00:02 2023-12-26T21:40:29.659 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:40:29.761 INFO:teuthology.orchestra.run.smithi059.stdout:(2/3): kernel-modules-4.18.0-529.el8.x86_64.rpm 22 MB/s | 36 MB 00:01 2023-12-26T21:40:30.097 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:40:30.097 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:40:30.269 INFO:teuthology.orchestra.run.smithi059.stdout:(3/3): kernel-core-4.18.0-529.el8.x86_64.rpm 20 MB/s | 43 MB 00:02 2023-12-26T21:40:30.270 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:40:30.270 INFO:teuthology.orchestra.run.smithi059.stdout:Total 35 MB/s | 89 MB 00:02 2023-12-26T21:40:30.983 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:40:31.327 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:40:31.353 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:40:31.423 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:40:31.423 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:40:32.716 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:40:32.741 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:40:34.143 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:40:34.724 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-26T21:40:35.667 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:40:36.380 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-26T21:40:38.375 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-26T21:40:38.766 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-26T21:40:40.113 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 1/3 2023-12-26T21:40:40.479 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-26T21:40:42.194 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-26T21:40:42.489 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:40:44.157 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 2/3 2023-12-26T21:40:44.430 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : kernel-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:04.396 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:04.421 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:04.762 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:04.762 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-26T21:41:04.762 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-26T21:41:05.584 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:05.585 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:41:05.585 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:41:05.585 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-26T21:41:05.585 INFO:teuthology.orchestra.run.smithi076.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-26T21:41:05.586 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:41:05.586 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:41:05.729 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2023-12-26T21:41:05.968 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-26T21:41:05.969 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-26T21:41:05.969 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-26T21:41:05.969 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-26T21:41:05.969 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-26T21:41:05.969 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-12-26T21:41:05.970 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi076.front.sepia.ceph.com, path=None, version=distro) 2023-12-26T21:41:05.970 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q kernel | sort -rV | head -n 1 2023-12-26T21:41:06.180 INFO:teuthology.orchestra.run.smithi076.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-26T21:41:06.180 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-26T21:41:06.180 DEBUG:teuthology.orchestra.run.smithi076:> sudo rpm -qi grub2-tools 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Name : grub2-tools 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Epoch : 1 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Version : 2.02 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Release : 129.el8 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Architecture: x86_64 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Group : System Environment/Base 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:Size : 9265586 2023-12-26T21:41:06.214 INFO:teuthology.orchestra.run.smithi076.stdout:License : GPLv3+ 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Relocations : (not relocatable) 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Packager : CentOS BuildSystem 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Vendor : CentOS 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Summary : Support tools for GRUB. 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:Description : 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-26T21:41:06.215 INFO:teuthology.orchestra.run.smithi076.stdout:hardware devices. 2023-12-26T21:41:06.216 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:41:06.216 INFO:teuthology.orchestra.run.smithi076.stdout:This subpackage provides tools for support of all platforms. 2023-12-26T21:41:06.216 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-26T21:41:06.217 INFO:teuthology.task.kernel:Updating grub on smithi076 to boot 4.18.0-529.el8.x86_64 2023-12-26T21:41:06.217 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-26T21:41:07.887 INFO:teuthology.orchestra.run.smithi076.stderr:Generating grub configuration file ... 2023-12-26T21:41:07.944 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-core-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:07.973 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:08.407 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: kernel-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:08.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-4.18.0-529.el8.x86_64 1/3 2023-12-26T21:41:08.408 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-core-4.18.0-529.el8.x86_64 2/3 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : kernel-modules-4.18.0-529.el8.x86_64 3/3 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-4.18.0-529.el8.x86_64 kernel-core-4.18.0-529.el8.x86_64 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout: kernel-modules-4.18.0-529.el8.x86_64 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:41:09.225 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:41:09.383 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2023-12-26T21:41:09.656 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-26T21:41:09.656 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-26T21:41:09.656 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-529.el8.x86_64 2023-12-26T21:41:09.656 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-12-26T21:41:09.656 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-12-26T21:41:09.656 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-12-26T21:41:09.656 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi059.front.sepia.ceph.com, path=None, version=distro) 2023-12-26T21:41:09.656 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q kernel | sort -rV | head -n 1 2023-12-26T21:41:09.877 INFO:teuthology.orchestra.run.smithi059.stdout:kernel-4.18.0-529.el8.x86_64 2023-12-26T21:41:09.877 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-529.el8.x86_64 2023-12-26T21:41:09.877 DEBUG:teuthology.orchestra.run.smithi059:> sudo rpm -qi grub2-tools 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Name : grub2-tools 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Epoch : 1 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Version : 2.02 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Release : 129.el8 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Architecture: x86_64 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Group : System Environment/Base 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Size : 9265586 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:License : GPLv3+ 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-12-26T21:41:09.912 INFO:teuthology.orchestra.run.smithi059.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Build Host : kbuilder.bsys.centos.org 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Relocations : (not relocatable) 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Packager : CentOS BuildSystem 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Vendor : CentOS 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:URL : http://www.gnu.org/software/grub/ 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Summary : Support tools for GRUB. 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:Description : 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:customizable bootloader with modular architecture. It supports a rich 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:variety of kernel formats, file systems, computer architectures and 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:hardware devices. 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:41:09.913 INFO:teuthology.orchestra.run.smithi059.stdout:This subpackage provides tools for support of all platforms. 2023-12-26T21:41:09.914 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-12-26T21:41:09.914 INFO:teuthology.task.kernel:Updating grub on smithi059 to boot 4.18.0-529.el8.x86_64 2023-12-26T21:41:09.915 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-12-26T21:41:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:done 2023-12-26T21:41:10.367 DEBUG:teuthology.orchestra.run.smithi076:> mktemp 2023-12-26T21:41:10.398 INFO:teuthology.orchestra.run.smithi076.stdout:/tmp/tmp.SBFS4g2p7t 2023-12-26T21:41:10.399 DEBUG:teuthology.orchestra.run.smithi076:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.SBFS4g2p7t 2023-12-26T21:41:10.473 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 0666 /tmp/tmp.SBFS4g2p7t 2023-12-26T21:41:10.615 DEBUG:teuthology.orchestra.remote:smithi076:/tmp/tmp.SBFS4g2p7t is 6KB 2023-12-26T21:41:10.661 DEBUG:teuthology.orchestra.run.smithi076:> rm -fr /tmp/tmp.SBFS4g2p7t 2023-12-26T21:41:10.679 DEBUG:teuthology.orchestra.run.smithi076:> sudo /bin/ls /boot/loader/entries || true 2023-12-26T21:41:10.755 INFO:teuthology.orchestra.run.smithi076.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-26T21:41:10.755 INFO:teuthology.orchestra.run.smithi076.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-26T21:41:10.755 INFO:teuthology.orchestra.run.smithi076.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-26T21:41:10.755 INFO:teuthology.orchestra.run.smithi076.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-26T21:41:10.756 DEBUG:teuthology.orchestra.run.smithi076:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-26T21:41:10.938 DEBUG:teuthology.orchestra.run.smithi076:> sudo shutdown -r now 2023-12-26T21:41:11.240 INFO:teuthology.orchestra.run.smithi059.stderr:Generating grub configuration file ... 2023-12-26T21:41:13.667 INFO:teuthology.orchestra.run.smithi059.stderr:done 2023-12-26T21:41:13.668 DEBUG:teuthology.orchestra.run.smithi059:> mktemp 2023-12-26T21:41:13.706 INFO:teuthology.orchestra.run.smithi059.stdout:/tmp/tmp.DIyiSq60z4 2023-12-26T21:41:13.706 DEBUG:teuthology.orchestra.run.smithi059:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.DIyiSq60z4 2023-12-26T21:41:13.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 0666 /tmp/tmp.DIyiSq60z4 2023-12-26T21:41:13.923 DEBUG:teuthology.orchestra.remote:smithi059:/tmp/tmp.DIyiSq60z4 is 6KB 2023-12-26T21:41:13.970 DEBUG:teuthology.orchestra.run.smithi059:> rm -fr /tmp/tmp.DIyiSq60z4 2023-12-26T21:41:13.989 DEBUG:teuthology.orchestra.run.smithi059:> sudo /bin/ls /boot/loader/entries || true 2023-12-26T21:41:14.070 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-12-26T21:41:14.070 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-12-26T21:41:14.071 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-12-26T21:41:14.071 INFO:teuthology.orchestra.run.smithi059.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64.conf 2023-12-26T21:41:14.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-529.el8.x86_64 2023-12-26T21:41:14.232 DEBUG:teuthology.orchestra.run.smithi059:> sudo shutdown -r now 2023-12-26T21:41:40.966 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-12-26T21:41:40.967 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:41:40.970 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:41:44.237 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-12-26T21:41:44.238 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-26T21:41:44.238 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:41:59.445 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-26T21:42:02.769 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.59 2023-12-26T21:42:08.450 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:42:08.450 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:42:11.505 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.76 2023-12-26T21:42:11.771 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-26T21:42:11.771 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:42:23.613 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:42:23.614 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:43:11.822 DEBUG:teuthology.orchestra.remote:timed out 2023-12-26T21:43:23.625 DEBUG:teuthology.orchestra.remote:timed out 2023-12-26T21:43:23.823 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-26T21:43:23.824 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:43:24.174 DEBUG:teuthology.orchestra.run.smithi059:> true 2023-12-26T21:43:24.478 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-26T21:43:24.478 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-529.el8.x86_64"... 2023-12-26T21:43:24.479 DEBUG:teuthology.orchestra.run.smithi059:> uname -r 2023-12-26T21:43:24.498 INFO:teuthology.orchestra.run.smithi059.stdout:4.18.0-529.el8.x86_64 2023-12-26T21:43:24.498 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-26T21:43:24.498 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-26T21:43:24.498 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-26T21:43:25.500 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-12-26T21:43:25.500 DEBUG:teuthology.orchestra.run.smithi059:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-26T21:43:25.571 INFO:teuthology.orchestra.run.smithi059.stdout:ttyS1 2023-12-26T21:43:25.587 DEBUG:teuthology.parallel:result is None 2023-12-26T21:43:38.627 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:43:38.627 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:43:38.947 DEBUG:teuthology.orchestra.run.smithi076:> true 2023-12-26T21:43:39.237 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:43:39.237 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-529.el8.x86_64"... 2023-12-26T21:43:39.238 DEBUG:teuthology.orchestra.run.smithi076:> uname -r 2023-12-26T21:43:39.299 INFO:teuthology.orchestra.run.smithi076.stdout:4.18.0-529.el8.x86_64 2023-12-26T21:43:39.299 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-529.el8.x86_64 vs 4.18.0-529.el8.x86_64 2023-12-26T21:43:39.299 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-12-26T21:43:39.299 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-12-26T21:43:40.300 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-12-26T21:43:40.300 DEBUG:teuthology.orchestra.run.smithi076:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-12-26T21:43:40.383 INFO:teuthology.orchestra.run.smithi076.stdout:ttyS1 2023-12-26T21:43:40.397 DEBUG:teuthology.parallel:result is None 2023-12-26T21:43:40.397 INFO:teuthology.run_tasks:Running task internal.base... 2023-12-26T21:43:40.409 INFO:teuthology.task.internal:Creating test directory... 2023-12-26T21:43:40.410 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-26T21:43:40.412 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-12-26T21:43:40.434 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-12-26T21:43:40.452 INFO:teuthology.run_tasks:Running task internal.archive... 2023-12-26T21:43:40.497 INFO:teuthology.task.internal:Creating archive directory... 2023-12-26T21:43:40.498 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-26T21:43:40.500 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-12-26T21:43:40.545 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-12-26T21:43:40.554 INFO:teuthology.task.internal:Enabling coredump saving... 2023-12-26T21:43:40.554 DEBUG:teuthology.orchestra.run.smithi059:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-26T21:43:40.580 DEBUG:teuthology.orchestra.run.smithi076:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-12-26T21:43:40.614 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-26T21:43:40.617 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-26T21:43:40.626 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-26T21:43:40.629 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-12-26T21:43:40.630 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-12-26T21:43:40.637 INFO:teuthology.task.internal:Configuring sudo... 2023-12-26T21:43:40.638 DEBUG:teuthology.orchestra.run.smithi059:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-26T21:43:40.670 DEBUG:teuthology.orchestra.run.smithi076:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-12-26T21:43:40.701 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-12-26T21:43:40.716 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-12-26T21:43:40.716 DEBUG:teuthology.orchestra.run.smithi059:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-26T21:43:40.742 DEBUG:teuthology.orchestra.run.smithi076:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-12-26T21:43:40.761 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-26T21:43:40.846 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-26T21:43:40.913 DEBUG:teuthology.orchestra.run.smithi059:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-26T21:43:40.956 DEBUG:teuthology.orchestra.run.smithi059:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-26T21:43:41.002 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:43:41.002 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-26T21:43:41.070 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-26T21:43:41.118 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-12-26T21:43:41.168 DEBUG:teuthology.orchestra.run.smithi076:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-26T21:43:41.255 DEBUG:teuthology.orchestra.run.smithi076:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-12-26T21:43:41.298 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:43:41.298 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-12-26T21:43:41.366 DEBUG:teuthology.orchestra.run.smithi059:> sudo service rsyslog restart 2023-12-26T21:43:41.368 DEBUG:teuthology.orchestra.run.smithi076:> sudo service rsyslog restart 2023-12-26T21:43:41.440 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-26T21:43:41.484 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-26T21:43:41.812 INFO:teuthology.run_tasks:Running task internal.timer... 2023-12-26T21:43:41.821 INFO:teuthology.task.internal:Starting timer... 2023-12-26T21:43:41.821 INFO:teuthology.run_tasks:Running task pcp... 2023-12-26T21:43:41.863 INFO:teuthology.run_tasks:Running task selinux... 2023-12-26T21:43:41.905 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-12-26T21:43:41.906 DEBUG:teuthology.orchestra.run.smithi059:> sudo service auditd rotate 2023-12-26T21:43:41.984 INFO:teuthology.orchestra.run.smithi059.stdout:Rotating logs: 2023-12-26T21:43:41.985 DEBUG:teuthology.orchestra.run.smithi076:> sudo service auditd rotate 2023-12-26T21:43:42.051 INFO:teuthology.orchestra.run.smithi076.stdout:Rotating logs: 2023-12-26T21:43:42.053 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-12-26T21:43:42.053 DEBUG:teuthology.orchestra.run.smithi059:> /usr/sbin/getenforce 2023-12-26T21:43:42.100 INFO:teuthology.orchestra.run.smithi059.stdout:Permissive 2023-12-26T21:43:42.100 DEBUG:teuthology.orchestra.run.smithi076:> /usr/sbin/getenforce 2023-12-26T21:43:42.137 INFO:teuthology.orchestra.run.smithi076.stdout:Permissive 2023-12-26T21:43:42.138 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi059.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi076.front.sepia.ceph.com': 'permissive'} 2023-12-26T21:43:42.138 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-26T21:43:42.167 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T21:43:42.168 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-26T21:43:42.205 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T21:43:42.205 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-12-26T21:43:42.206 DEBUG:teuthology.orchestra.run.smithi059:> sudo /usr/sbin/setenforce permissive 2023-12-26T21:43:42.236 DEBUG:teuthology.orchestra.run.smithi076:> sudo /usr/sbin/setenforce permissive 2023-12-26T21:43:42.274 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-12-26T21:43:42.283 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-12-26T21:43:42.303 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-12-26T21:43:42.307 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi059.front.sepia.ceph.com,smithi076.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-12-26T21:51:21.787 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi059.front.sepia.ceph.com'), Remote(name='ubuntu@smithi076.front.sepia.ceph.com')] 2023-12-26T21:51:21.788 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-26T21:51:21.789 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi059.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:51:21.858 DEBUG:teuthology.orchestra.run.smithi059:> true 2023-12-26T21:51:21.886 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi059.front.sepia.ceph.com' 2023-12-26T21:51:21.886 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:51:21.887 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi076.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-12-26T21:51:21.954 DEBUG:teuthology.orchestra.run.smithi076:> true 2023-12-26T21:51:22.023 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi076.front.sepia.ceph.com' 2023-12-26T21:51:22.024 INFO:teuthology.run_tasks:Running task clock... 2023-12-26T21:51:22.035 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-12-26T21:51:22.035 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-26T21:51:22.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-26T21:51:22.039 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-12-26T21:51:22.039 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-26T21:51:22.074 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-26T21:51:22.088 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-26T21:51:22.095 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-12-26T21:51:22.108 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-12-26T21:51:22.166 INFO:teuthology.orchestra.run.smithi059.stderr:sudo: ntpd: command not found 2023-12-26T21:51:22.173 INFO:teuthology.orchestra.run.smithi076.stderr:sudo: ntpd: command not found 2023-12-26T21:51:22.178 INFO:teuthology.orchestra.run.smithi059.stdout:506 Cannot talk to daemon 2023-12-26T21:51:22.184 INFO:teuthology.orchestra.run.smithi076.stdout:506 Cannot talk to daemon 2023-12-26T21:51:22.193 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-26T21:51:22.198 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-12-26T21:51:22.206 INFO:teuthology.orchestra.run.smithi059.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-26T21:51:22.211 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-12-26T21:51:22.274 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2023-12-26T21:51:22.277 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-26T21:51:22.277 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2023-12-26T21:51:22.277 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.277 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.277 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.277 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.282 INFO:teuthology.orchestra.run.smithi076.stderr:bash: ntpq: command not found 2023-12-26T21:51:22.286 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-26T21:51:22.286 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2023-12-26T21:51:22.286 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.287 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.287 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.287 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T21:51:22.288 INFO:teuthology.run_tasks:Running task pexec... 2023-12-26T21:51:22.298 INFO:teuthology.task.pexec:Executing custom commands... 2023-12-26T21:51:22.298 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi059.front.sepia.ceph.com 2023-12-26T21:51:22.299 DEBUG:teuthology.orchestra.run.smithi059:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-26T21:51:22.299 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi076.front.sepia.ceph.com 2023-12-26T21:51:22.299 DEBUG:teuthology.orchestra.run.smithi076:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-12-26T21:51:23.202 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:46 ago on Tue 26 Dec 2023 09:49:37 PM UTC. 2023-12-26T21:51:23.230 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:37 ago on Tue 26 Dec 2023 09:49:46 PM UTC. 2023-12-26T21:51:25.353 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout:Resetting modules: 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout: container-tools 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:51:25.354 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:25.602 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:51:25.606 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:51:25.607 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:51:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-26T21:51:25.607 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:51:25.607 INFO:teuthology.orchestra.run.smithi059.stdout:Resetting modules: 2023-12-26T21:51:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools 2023-12-26T21:51:25.607 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:25.608 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:51:25.608 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:51:25.608 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:25.884 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:51:26.544 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:49 ago on Tue 26 Dec 2023 09:49:37 PM UTC. 2023-12-26T21:51:26.858 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:40 ago on Tue 26 Dec 2023 09:49:46 PM UTC. 2023-12-26T21:51:28.674 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:51:28.861 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:51:28.861 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout:Installing group/module packages: 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-26T21:51:28.862 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:Installing weak dependencies: 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:Installing module profiles: 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: container-tools/common 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:Enabling module streams: 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: container-tools rhel8 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout:Install 13 Packages 2023-12-26T21:51:28.863 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:28.864 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 27 M 2023-12-26T21:51:28.864 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 89 M 2023-12-26T21:51:28.864 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:51:29.136 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:51:29.237 INFO:teuthology.orchestra.run.smithi076.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.2 MB/s | 750 kB 00:00 2023-12-26T21:51:29.279 INFO:teuthology.orchestra.run.smithi076.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.7 MB/s | 1.0 MB 00:00 2023-12-26T21:51:29.304 INFO:teuthology.orchestra.run.smithi076.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 566 kB/s | 38 kB 00:00 2023-12-26T21:51:29.337 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout:Installing group/module packages: 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:51:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:Installing weak dependencies: 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:Installing module profiles: 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools/common 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:Enabling module streams: 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: container-tools rhel8 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2023-12-26T21:51:29.339 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:29.340 INFO:teuthology.orchestra.run.smithi076.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.0 MB/s | 241 kB 00:00 2023-12-26T21:51:29.340 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 27 M 2023-12-26T21:51:29.341 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 89 M 2023-12-26T21:51:29.341 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:51:29.397 INFO:teuthology.orchestra.run.smithi076.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2023-12-26T21:51:29.455 INFO:teuthology.orchestra.run.smithi076.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 423 kB/s | 25 kB 00:00 2023-12-26T21:51:29.537 INFO:teuthology.orchestra.run.smithi076.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 17 MB/s | 3.9 MB 00:00 2023-12-26T21:51:29.565 INFO:teuthology.orchestra.run.smithi076.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 861 kB/s | 94 kB 00:00 2023-12-26T21:51:29.715 INFO:teuthology.orchestra.run.smithi076.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-12-26T21:51:29.773 INFO:teuthology.orchestra.run.smithi076.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 903 kB/s | 53 kB 00:00 2023-12-26T21:51:29.824 INFO:teuthology.orchestra.run.smithi076.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 9.8 MB/s | 2.5 MB 00:00 2023-12-26T21:51:29.849 INFO:teuthology.orchestra.run.smithi076.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 544 kB/s | 41 kB 00:00 2023-12-26T21:51:29.856 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.2 MB/s | 750 kB 00:00 2023-12-26T21:51:29.913 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 2.6 MB/s | 1.0 MB 00:00 2023-12-26T21:51:29.948 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 413 kB/s | 38 kB 00:00 2023-12-26T21:51:29.957 INFO:teuthology.orchestra.run.smithi076.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 19 MB/s | 8.1 MB 00:00 2023-12-26T21:51:29.958 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:51:29.958 INFO:teuthology.orchestra.run.smithi076.stdout:Total 24 MB/s | 27 MB 00:01 2023-12-26T21:51:29.965 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 4.5 MB/s | 241 kB 00:00 2023-12-26T21:51:30.024 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2023-12-26T21:51:30.082 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 426 kB/s | 25 kB 00:00 2023-12-26T21:51:30.183 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 17 MB/s | 3.9 MB 00:00 2023-12-26T21:51:30.208 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 749 kB/s | 94 kB 00:00 2023-12-26T21:51:30.224 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:51:30.245 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:51:30.245 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:51:30.308 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 12 MB/s | 9.6 MB 00:00 2023-12-26T21:51:30.367 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 16 MB/s | 2.5 MB 00:00 2023-12-26T21:51:30.417 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 484 kB/s | 53 kB 00:00 2023-12-26T21:51:30.434 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 612 kB/s | 41 kB 00:00 2023-12-26T21:51:30.501 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:51:30.504 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:51:30.576 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 21 MB/s | 8.1 MB 00:00 2023-12-26T21:51:30.576 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:51:30.576 INFO:teuthology.orchestra.run.smithi059.stdout:Total 21 MB/s | 27 MB 00:01 2023-12-26T21:51:30.853 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:51:30.876 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:51:30.876 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:51:31.152 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:51:31.155 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:51:31.219 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:51:31.465 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-26T21:51:31.627 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-26T21:51:31.816 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-26T21:51:31.916 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:51:32.029 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-26T21:51:32.140 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-12-26T21:51:32.343 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-12-26T21:51:32.521 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-12-26T21:51:32.543 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-26T21:51:32.782 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-26T21:51:33.366 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-12-26T21:51:33.569 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-26T21:51:33.800 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-26T21:51:34.012 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-26T21:51:34.281 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-26T21:51:34.416 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-26T21:51:34.653 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-12-26T21:51:34.730 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-26T21:51:34.866 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-12-26T21:51:35.138 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-12-26T21:51:35.551 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-12-26T21:51:36.040 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-26T21:51:36.248 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-26T21:51:36.451 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-26T21:51:36.495 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-26T21:51:36.723 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-26T21:51:36.871 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-26T21:51:36.891 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-12-26T21:51:37.364 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-12-26T21:51:37.365 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-12-26T21:51:37.365 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-12-26T21:51:37.365 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:37.688 INFO:teuthology.orchestra.run.smithi076.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:51:37.689 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:51:37.943 DEBUG:teuthology.parallel:result is None 2023-12-26T21:51:38.215 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-12-26T21:51:38.215 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:38.215 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:51:38.215 INFO:teuthology.orchestra.run.smithi059.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: yajl-2.1.0-12.el8.x86_64 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:51:38.216 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:51:38.496 DEBUG:teuthology.parallel:result is None 2023-12-26T21:51:38.496 INFO:teuthology.run_tasks:Running task install... 2023-12-26T21:51:38.504 DEBUG:teuthology.task.install:project ceph 2023-12-26T21:51:38.504 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '5b5a60d2d510bfaa9213c41e32227fb008245a30'}} 2023-12-26T21:51:38.504 DEBUG:teuthology.task.install:config {'branch': 'pacific', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '5b5a60d2d510bfaa9213c41e32227fb008245a30'} 2023-12-26T21:51:38.504 INFO:teuthology.task.install:Using flavor: default 2023-12-26T21:51:38.510 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-12-26T21:51:38.510 INFO:teuthology.task.install:extra packages: [] 2023-12-26T21:51:38.510 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5b5a60d2d510bfaa9213c41e32227fb008245a30', 'tag': None, 'wait_for_package': False} 2023-12-26T21:51:38.511 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:51:38.511 INFO:teuthology.packaging:ref: None 2023-12-26T21:51:38.511 INFO:teuthology.packaging:tag: None 2023-12-26T21:51:38.511 INFO:teuthology.packaging:branch: pacific 2023-12-26T21:51:38.511 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:51:38.511 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-12-26T21:51:38.513 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'pacific', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '5b5a60d2d510bfaa9213c41e32227fb008245a30', 'tag': None, 'wait_for_package': False} 2023-12-26T21:51:38.513 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:51:38.513 INFO:teuthology.packaging:ref: None 2023-12-26T21:51:38.513 INFO:teuthology.packaging:tag: None 2023-12-26T21:51:38.513 INFO:teuthology.packaging:branch: pacific 2023-12-26T21:51:38.513 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:51:38.513 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-12-26T21:51:38.723 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/ 2023-12-26T21:51:38.723 INFO:teuthology.task.install.rpm:Package version is 16.2.14-383.g2c2e7b98 2023-12-26T21:51:38.731 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/ 2023-12-26T21:51:38.732 INFO:teuthology.task.install.rpm:Package version is 16.2.14-383.g2c2e7b98 2023-12-26T21:51:38.891 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-26T21:51:38.891 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:51:38.891 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-26T21:51:38.899 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/pacific/2c2e7b98dd86be9dcfdd9b3c5d0eb1eab488446c/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-12-26T21:51:38.899 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:51:38.899 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-12-26T21:51:38.920 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-26T21:51:38.920 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:51:38.920 INFO:teuthology.packaging:ref: None 2023-12-26T21:51:38.921 INFO:teuthology.packaging:tag: None 2023-12-26T21:51:38.921 INFO:teuthology.packaging:branch: pacific 2023-12-26T21:51:38.921 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:51:38.921 DEBUG:teuthology.orchestra.run.smithi076:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-26T21:51:38.930 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-12-26T21:51:38.930 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:51:38.931 INFO:teuthology.packaging:ref: None 2023-12-26T21:51:38.931 INFO:teuthology.packaging:tag: None 2023-12-26T21:51:38.931 INFO:teuthology.packaging:branch: pacific 2023-12-26T21:51:38.931 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:51:38.931 DEBUG:teuthology.orchestra.run.smithi059:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/pacific/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-12-26T21:51:38.990 DEBUG:teuthology.orchestra.run.smithi076:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-26T21:51:39.001 DEBUG:teuthology.orchestra.run.smithi059:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-12-26T21:51:39.066 DEBUG:teuthology.orchestra.run.smithi076:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-26T21:51:39.079 DEBUG:teuthology.orchestra.run.smithi059:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-12-26T21:51:39.133 INFO:teuthology.orchestra.run.smithi076.stdout:check_obsoletes = 1 2023-12-26T21:51:39.134 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum clean all 2023-12-26T21:51:39.148 INFO:teuthology.orchestra.run.smithi059.stdout:check_obsoletes = 1 2023-12-26T21:51:39.149 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum clean all 2023-12-26T21:51:39.407 INFO:teuthology.orchestra.run.smithi076.stdout:76 files removed 2023-12-26T21:51:39.433 INFO:teuthology.orchestra.run.smithi059.stdout:76 files removed 2023-12-26T21:51:39.435 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-radosgw 2023-12-26T21:51:39.462 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-radosgw 2023-12-26T21:51:40.100 INFO:teuthology.orchestra.run.smithi076.stdout:ceph packages for x86_64 296 kB/s | 76 kB 00:00 2023-12-26T21:51:40.209 INFO:teuthology.orchestra.run.smithi059.stdout:ceph packages for x86_64 241 kB/s | 76 kB 00:00 2023-12-26T21:51:40.468 INFO:teuthology.orchestra.run.smithi076.stdout:ceph noarch packages 46 kB/s | 15 kB 00:00 2023-12-26T21:51:40.576 INFO:teuthology.orchestra.run.smithi059.stdout:ceph noarch packages 47 kB/s | 15 kB 00:00 2023-12-26T21:51:40.802 INFO:teuthology.orchestra.run.smithi076.stdout:ceph source packages 4.7 kB/s | 1.4 kB 00:00 2023-12-26T21:51:40.902 INFO:teuthology.orchestra.run.smithi059.stdout:ceph source packages 4.9 kB/s | 1.4 kB 00:00 2023-12-26T21:51:42.819 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:01 2023-12-26T21:51:43.019 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - AppStream 17 MB/s | 35 MB 00:02 2023-12-26T21:51:53.194 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 57 MB 00:04 2023-12-26T21:51:53.879 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - BaseOS 14 MB/s | 57 MB 00:04 2023-12-26T21:52:01.494 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - Extras 84 kB/s | 18 kB 00:00 2023-12-26T21:52:01.845 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - Extras common packages 40 kB/s | 6.9 kB 00:00 2023-12-26T21:52:02.979 INFO:teuthology.orchestra.run.smithi076.stdout:CentOS Stream 8 - PowerTools 6.7 MB/s | 6.6 MB 00:00 2023-12-26T21:52:03.861 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras 13 kB/s | 18 kB 00:01 2023-12-26T21:52:04.378 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - Extras common packages 22 kB/s | 6.9 kB 00:00 2023-12-26T21:52:04.431 INFO:teuthology.orchestra.run.smithi076.stdout:Copr repo for python3-asyncssh owned by ceph 22 kB/s | 3.5 kB 00:00 2023-12-26T21:52:05.455 INFO:teuthology.orchestra.run.smithi059.stdout:CentOS Stream 8 - PowerTools 7.3 MB/s | 6.6 MB 00:00 2023-12-26T21:52:05.484 INFO:teuthology.orchestra.run.smithi076.stdout:Extra Packages for Enterprise Linux 18 MB/s | 16 MB 00:00 2023-12-26T21:52:06.957 INFO:teuthology.orchestra.run.smithi059.stdout:Copr repo for python3-asyncssh owned by ceph 29 kB/s | 3.5 kB 00:00 2023-12-26T21:52:09.399 INFO:teuthology.orchestra.run.smithi076.stdout:lab-extras 465 kB/s | 24 kB 00:00 2023-12-26T21:52:10.157 INFO:teuthology.orchestra.run.smithi059.stdout:Extra Packages for Enterprise Linux 5.3 MB/s | 16 MB 00:03 2023-12-26T21:52:14.054 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 12 M 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrading: 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout: librados2 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 3.7 M 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 4.2 M 2023-12-26T21:52:14.057 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 6.1 M 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 24 M 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 26 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 825 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 494 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 3.8 M 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-26T21:52:14.058 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 45 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 99 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 215 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 389 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 369 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 114 k 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout:Install 21 Packages 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout:Upgrade 2 Packages 2023-12-26T21:52:14.059 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:14.060 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 58 M 2023-12-26T21:52:14.060 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:52:14.308 INFO:teuthology.orchestra.run.smithi059.stdout:lab-extras 362 kB/s | 24 kB 00:00 2023-12-26T21:52:15.434 INFO:teuthology.orchestra.run.smithi076.stdout:(1/23): ceph-base-16.2.14-383.g2c2e7b98.el8.x86 7.5 MB/s | 6.1 MB 00:00 2023-12-26T21:52:15.501 INFO:teuthology.orchestra.run.smithi076.stdout:(2/23): ceph-selinux-16.2.14-383.g2c2e7b98.el8. 385 kB/s | 26 kB 00:00 2023-12-26T21:52:15.610 INFO:teuthology.orchestra.run.smithi076.stdout:(3/23): libcephfs2-16.2.14-383.g2c2e7b98.el8.x8 7.4 MB/s | 825 kB 00:00 2023-12-26T21:52:15.718 INFO:teuthology.orchestra.run.smithi076.stdout:(4/23): libradosstriper1-16.2.14-383.g2c2e7b98. 4.5 MB/s | 494 kB 00:00 2023-12-26T21:52:16.077 INFO:teuthology.orchestra.run.smithi076.stdout:(5/23): librgw2-16.2.14-383.g2c2e7b98.el8.x86_6 11 MB/s | 3.8 MB 00:00 2023-12-26T21:52:16.144 INFO:teuthology.orchestra.run.smithi076.stdout:(6/23): python3-ceph-argparse-16.2.14-383.g2c2e 678 kB/s | 45 kB 00:00 2023-12-26T21:52:16.219 INFO:teuthology.orchestra.run.smithi076.stdout:(7/23): python3-ceph-common-16.2.14-383.g2c2e7b 1.3 MB/s | 99 kB 00:00 2023-12-26T21:52:16.353 INFO:teuthology.orchestra.run.smithi076.stdout:(8/23): ceph-radosgw-16.2.14-383.g2c2e7b98.el8. 6.7 MB/s | 12 MB 00:01 2023-12-26T21:52:16.378 INFO:teuthology.orchestra.run.smithi076.stdout:(9/23): python3-cephfs-16.2.14-383.g2c2e7b98.el 1.3 MB/s | 215 kB 00:00 2023-12-26T21:52:16.437 INFO:teuthology.orchestra.run.smithi076.stdout:(10/23): python3-rados-16.2.14-383.g2c2e7b98.el 4.6 MB/s | 389 kB 00:00 2023-12-26T21:52:16.471 INFO:teuthology.orchestra.run.smithi076.stdout:(11/23): python3-rbd-16.2.14-383.g2c2e7b98.el8. 3.9 MB/s | 369 kB 00:00 2023-12-26T21:52:16.504 INFO:teuthology.orchestra.run.smithi076.stdout:(12/23): python3-rgw-16.2.14-383.g2c2e7b98.el8. 1.7 MB/s | 114 kB 00:00 2023-12-26T21:52:16.730 INFO:teuthology.orchestra.run.smithi076.stdout:(13/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.1 MB/s | 259 kB 00:00 2023-12-26T21:52:16.755 INFO:teuthology.orchestra.run.smithi076.stdout:(14/23): librdkafka-1.6.1-1.el8.x86_64.rpm 2.3 MB/s | 677 kB 00:00 2023-12-26T21:52:16.789 INFO:teuthology.orchestra.run.smithi076.stdout:(15/23): python3-prettytable-0.7.2-14.el8.noarc 748 kB/s | 44 kB 00:00 2023-12-26T21:52:16.939 INFO:teuthology.orchestra.run.smithi076.stdout:(16/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 255 kB/s | 47 kB 00:00 2023-12-26T21:52:17.348 INFO:teuthology.orchestra.run.smithi076.stdout:(17/23): ceph-common-16.2.14-383.g2c2e7b98.el8. 8.9 MB/s | 24 MB 00:02 2023-12-26T21:52:17.465 INFO:teuthology.orchestra.run.smithi076.stdout:(18/23): gperftools-libs-2.7-9.el8.x86_64.rpm 453 kB/s | 306 kB 00:00 2023-12-26T21:52:17.573 INFO:teuthology.orchestra.run.smithi076.stdout:(19/23): libunwind-1.3.1-3.el8.x86_64.rpm 693 kB/s | 75 kB 00:00 2023-12-26T21:52:17.682 INFO:teuthology.orchestra.run.smithi076.stdout:(20/23): leveldb-1.22-1.el8.x86_64.rpm 244 kB/s | 181 kB 00:00 2023-12-26T21:52:17.765 INFO:teuthology.orchestra.run.smithi076.stdout:(21/23): liboath-2.6.2-3.el8.x86_64.rpm 141 kB/s | 59 kB 00:00 2023-12-26T21:52:17.932 INFO:teuthology.orchestra.run.smithi076.stdout:(22/23): librados2-16.2.14-383.g2c2e7b98.el8.x8 10 MB/s | 3.7 MB 00:00 2023-12-26T21:52:18.132 INFO:teuthology.orchestra.run.smithi076.stdout:(23/23): librbd1-16.2.14-383.g2c2e7b98.el8.x86_ 9.3 MB/s | 4.2 MB 00:00 2023-12-26T21:52:18.137 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:52:18.137 INFO:teuthology.orchestra.run.smithi076.stdout:Total 14 MB/s | 58 MB 00:04 2023-12-26T21:52:18.180 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:52:18.227 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:52:18.227 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:52:18.565 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:52:18.568 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:52:19.257 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 12 M 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrading: 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: librados2 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 3.7 M 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 4.2 M 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 6.1 M 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 24 M 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 26 k 2023-12-26T21:52:19.260 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 825 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 494 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2 x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 3.8 M 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 45 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 99 k 2023-12-26T21:52:19.261 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 215 k 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 389 k 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 369 k 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 114 k 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout:Install 21 Packages 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout:Upgrade 2 Packages 2023-12-26T21:52:19.262 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:52:19.263 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 58 M 2023-12-26T21:52:19.263 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:52:19.573 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:52:19.774 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-26T21:52:19.963 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-26T21:52:20.330 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-26T21:52:20.504 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/25 2023-12-26T21:52:20.599 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/25 2023-12-26T21:52:20.667 INFO:teuthology.orchestra.run.smithi059.stdout:(1/23): ceph-base-16.2.14-383.g2c2e7b98.el8.x86 7.2 MB/s | 6.1 MB 00:00 2023-12-26T21:52:20.774 INFO:teuthology.orchestra.run.smithi059.stdout:(2/23): ceph-selinux-16.2.14-383.g2c2e7b98.el8. 240 kB/s | 26 kB 00:00 2023-12-26T21:52:20.822 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 4/25 2023-12-26T21:52:21.012 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/25 2023-12-26T21:52:21.117 INFO:teuthology.orchestra.run.smithi059.stdout:(3/23): ceph-radosgw-16.2.14-383.g2c2e7b98.el8. 9.0 MB/s | 12 MB 00:01 2023-12-26T21:52:21.184 INFO:teuthology.orchestra.run.smithi059.stdout:(4/23): libradosstriper1-16.2.14-383.g2c2e7b98. 7.3 MB/s | 494 kB 00:00 2023-12-26T21:52:21.326 INFO:teuthology.orchestra.run.smithi059.stdout:(5/23): libcephfs2-16.2.14-383.g2c2e7b98.el8.x8 1.5 MB/s | 825 kB 00:00 2023-12-26T21:52:21.431 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/25 2023-12-26T21:52:21.452 INFO:teuthology.orchestra.run.smithi059.stdout:(6/23): python3-ceph-argparse-16.2.14-383.g2c2e 360 kB/s | 45 kB 00:00 2023-12-26T21:52:21.544 INFO:teuthology.orchestra.run.smithi059.stdout:(7/23): librgw2-16.2.14-383.g2c2e7b98.el8.x86_6 11 MB/s | 3.8 MB 00:00 2023-12-26T21:52:21.569 INFO:teuthology.orchestra.run.smithi059.stdout:(8/23): python3-ceph-common-16.2.14-383.g2c2e7b 846 kB/s | 99 kB 00:00 2023-12-26T21:52:21.594 INFO:teuthology.orchestra.run.smithi059.stdout:(9/23): python3-cephfs-16.2.14-383.g2c2e7b98.el 4.2 MB/s | 215 kB 00:00 2023-12-26T21:52:21.646 INFO:teuthology.orchestra.run.smithi076.stdout: Upgrading : librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/25 2023-12-26T21:52:21.661 INFO:teuthology.orchestra.run.smithi059.stdout:(10/23): python3-rbd-16.2.14-383.g2c2e7b98.el8. 5.4 MB/s | 369 kB 00:00 2023-12-26T21:52:21.696 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/25 2023-12-26T21:52:21.903 INFO:teuthology.orchestra.run.smithi059.stdout:(11/23): python3-rgw-16.2.14-383.g2c2e7b98.el8. 473 kB/s | 114 kB 00:00 2023-12-26T21:52:21.913 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-26T21:52:21.929 INFO:teuthology.orchestra.run.smithi059.stdout:(12/23): python3-rados-16.2.14-383.g2c2e7b98.el 1.1 MB/s | 389 kB 00:00 2023-12-26T21:52:22.179 INFO:teuthology.orchestra.run.smithi059.stdout:(13/23): ceph-common-16.2.14-383.g2c2e7b98.el8. 10 MB/s | 24 MB 00:02 2023-12-26T21:52:22.321 INFO:teuthology.orchestra.run.smithi059.stdout:(14/23): python3-prettytable-0.7.2-14.el8.noarc 311 kB/s | 44 kB 00:00 2023-12-26T21:52:22.367 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2023-12-26T21:52:22.388 INFO:teuthology.orchestra.run.smithi059.stdout:(15/23): lttng-ust-2.8.1-11.el8.x86_64.rpm 566 kB/s | 259 kB 00:00 2023-12-26T21:52:22.430 INFO:teuthology.orchestra.run.smithi059.stdout:(16/23): librdkafka-1.6.1-1.el8.x86_64.rpm 1.3 MB/s | 677 kB 00:00 2023-12-26T21:52:22.504 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 9/25 2023-12-26T21:52:22.513 INFO:teuthology.orchestra.run.smithi059.stdout:(17/23): librabbitmq-0.9.0-4.el8.x86_64.rpm 244 kB/s | 47 kB 00:00 2023-12-26T21:52:22.569 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 9/25 2023-12-26T21:52:22.743 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-26T21:52:22.939 INFO:teuthology.orchestra.run.smithi059.stdout:(18/23): leveldb-1.22-1.el8.x86_64.rpm 356 kB/s | 181 kB 00:00 2023-12-26T21:52:22.991 INFO:teuthology.orchestra.run.smithi059.stdout:(19/23): liboath-2.6.2-3.el8.x86_64.rpm 125 kB/s | 59 kB 00:00 2023-12-26T21:52:23.023 INFO:teuthology.orchestra.run.smithi059.stdout:(20/23): gperftools-libs-2.7-9.el8.x86_64.rpm 483 kB/s | 306 kB 00:00 2023-12-26T21:52:23.048 INFO:teuthology.orchestra.run.smithi059.stdout:(21/23): libunwind-1.3.1-3.el8.x86_64.rpm 688 kB/s | 75 kB 00:00 2023-12-26T21:52:23.152 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-argparse-2:16.2.14-383.g2c2e7b98.el8. 11/25 2023-12-26T21:52:23.316 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 12/25 2023-12-26T21:52:23.340 INFO:teuthology.orchestra.run.smithi059.stdout:(22/23): librbd1-16.2.14-383.g2c2e7b98.el8.x86_ 13 MB/s | 4.2 MB 00:00 2023-12-26T21:52:23.498 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 13/25 2023-12-26T21:52:23.721 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 14/25 2023-12-26T21:52:23.757 INFO:teuthology.orchestra.run.smithi059.stdout:(23/23): librados2-16.2.14-383.g2c2e7b98.el8.x8 4.9 MB/s | 3.7 MB 00:00 2023-12-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:52:23.762 INFO:teuthology.orchestra.run.smithi059.stdout:Total 13 MB/s | 58 MB 00:04 2023-12-26T21:52:23.809 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:52:23.846 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_6 15/25 2023-12-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:52:23.862 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:52:23.917 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_6 15/25 2023-12-26T21:52:24.122 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-26T21:52:24.236 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:52:24.240 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:52:24.312 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-26T21:52:24.526 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-26T21:52:24.713 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-ceph-common-2:16.2.14-383.g2c2e7b98.el8.x8 19/25 2023-12-26T21:52:25.345 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:52:25.579 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : liboath-2.6.2-3.el8.x86_64 1/25 2023-12-26T21:52:25.785 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-26T21:52:26.191 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 2/25 2023-12-26T21:52:26.410 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/25 2023-12-26T21:52:26.513 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/25 2023-12-26T21:52:26.954 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 4/25 2023-12-26T21:52:27.135 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/25 2023-12-26T21:52:27.192 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 20/25 2023-12-26T21:52:27.420 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 20/25 2023-12-26T21:52:27.574 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/25 2023-12-26T21:52:27.735 INFO:teuthology.orchestra.run.smithi059.stdout: Upgrading : librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/25 2023-12-26T21:52:27.786 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/25 2023-12-26T21:52:28.015 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/25 2023-12-26T21:52:28.135 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 20/25 2023-12-26T21:52:28.398 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 21/25 2023-12-26T21:52:28.502 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 21/25 2023-12-26T21:52:28.535 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/25 2023-12-26T21:52:28.687 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 22/25 2023-12-26T21:52:28.693 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 9/25 2023-12-26T21:52:28.760 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 9/25 2023-12-26T21:52:28.966 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/25 2023-12-26T21:52:29.141 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-argparse-2:16.2.14-383.g2c2e7b98.el8. 11/25 2023-12-26T21:52:29.338 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 12/25 2023-12-26T21:52:29.562 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 13/25 2023-12-26T21:52:29.754 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 14/25 2023-12-26T21:52:29.893 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_6 15/25 2023-12-26T21:52:29.966 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_6 15/25 2023-12-26T21:52:30.187 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 16/25 2023-12-26T21:52:30.385 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 17/25 2023-12-26T21:52:30.601 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 18/25 2023-12-26T21:52:30.827 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-ceph-common-2:16.2.14-383.g2c2e7b98.el8.x8 19/25 2023-12-26T21:52:33.489 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 20/25 2023-12-26T21:52:33.768 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 20/25 2023-12-26T21:52:34.530 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 20/25 2023-12-26T21:52:34.881 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 21/25 2023-12-26T21:52:34.991 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 21/25 2023-12-26T21:52:35.151 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 22/25 2023-12-26T21:52:46.732 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 22/25 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /sys 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /proc 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /dev 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /run 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /mnt 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /var/tmp 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /home 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /tmp 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout:skipping the directory /dev 2023-12-26T21:52:46.733 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:47.912 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 23/25 2023-12-26T21:52:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 23/25 2023-12-26T21:52:47.991 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-26T21:52:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-26T21:52:48.203 INFO:teuthology.orchestra.run.smithi076.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-26T21:52:48.843 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 4/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_6 6/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 7/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-argparse-2:16.2.14-383.g2c2e7b98.el8. 8/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-ceph-common-2:16.2.14-383.g2c2e7b98.el8.x8 9/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 10/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 11/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 12/25 2023-12-26T21:52:48.844 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 13/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-12-26T21:52:48.845 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-12-26T21:52:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 22/25 2023-12-26T21:52:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-26T21:52:48.846 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 24/25 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:Upgraded: 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout: librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout: librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.984 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: leveldb-1.22-1.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-argparse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.985 INFO:teuthology.orchestra.run.smithi076.stdout: python3-ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:49.986 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:52:50.250 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-test 2023-12-26T21:52:51.100 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:42 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:52:52.795 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 52 M 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:52:52.796 INFO:teuthology.orchestra.run.smithi076.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 174 k 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2023-12-26T21:52:52.797 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:52:52.798 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 53 M 2023-12-26T21:52:52.798 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 196 M 2023-12-26T21:52:52.798 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:52:53.208 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): jq-1.6-8.el8.x86_64.rpm 912 kB/s | 203 kB 00:00 2023-12-26T21:52:53.309 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): oniguruma-6.8.2-2.el8.x86_64.rpm 1.8 MB/s | 187 kB 00:00 2023-12-26T21:52:53.334 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): libcephsqlite-16.2.14-383.g2c2e7b98.el8. 499 kB/s | 174 kB 00:00 2023-12-26T21:52:53.393 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.8 MB/s | 323 kB 00:00 2023-12-26T21:52:53.518 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 384 kB/s | 71 kB 00:00 2023-12-26T21:52:54.059 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 22/25 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /sys 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /proc 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /run 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /mnt 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /var/tmp 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /home 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /tmp 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout:skipping the directory /dev 2023-12-26T21:52:54.060 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:52:55.320 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 23/25 2023-12-26T21:52:55.387 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 23/25 2023-12-26T21:52:55.387 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-26T21:52:55.485 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): ceph-test-16.2.14-383.g2c2e7b98.el8.x86_ 21 MB/s | 52 MB 00:02 2023-12-26T21:52:55.486 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:52:55.487 INFO:teuthology.orchestra.run.smithi076.stdout:Total 20 MB/s | 53 MB 00:02 2023-12-26T21:52:55.530 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:52:55.552 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:52:55.552 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:52:55.626 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 24/25 2023-12-26T21:52:55.626 INFO:teuthology.orchestra.run.smithi059.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-26T21:52:55.854 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:52:55.857 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:52:56.330 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:52:56.345 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 25/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 4/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_6 6/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 7/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-argparse-2:16.2.14-383.g2c2e7b98.el8. 8/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-ceph-common-2:16.2.14-383.g2c2e7b98.el8.x8 9/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 10/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 11/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 12/25 2023-12-26T21:52:56.346 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 13/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 14/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 15/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 16/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 17/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 18/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : leveldb-1.22-1.el8.x86_64 19/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 20/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 21/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 22/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 23/25 2023-12-26T21:52:56.347 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 24/25 2023-12-26T21:52:56.569 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-26T21:52:56.743 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-26T21:52:56.866 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-26T21:52:56.931 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-26T21:52:57.153 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-26T21:52:57.299 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/6 2023-12-26T21:52:57.599 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 25/25 2023-12-26T21:52:57.599 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:52:57.599 INFO:teuthology.orchestra.run.smithi059.stdout:Upgraded: 2023-12-26T21:52:57.599 INFO:teuthology.orchestra.run.smithi059.stdout: librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.599 INFO:teuthology.orchestra.run.smithi059.stdout: librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.599 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-radosgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-selinux-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: leveldb-1.22-1.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: liboath-2.6.2-3.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: libradosstriper1-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: librdkafka-1.6.1-1.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: librgw2-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: libunwind-1.3.1-3.el8.x86_64 2023-12-26T21:52:57.600 INFO:teuthology.orchestra.run.smithi059.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-argparse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-ceph-common-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:52:57.601 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:52:57.804 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-test 2023-12-26T21:52:58.690 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:44 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:00.509 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 52 M 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 174 k 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:00.510 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:53:00.511 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:00.511 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-12-26T21:53:00.511 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:00.511 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 53 M 2023-12-26T21:53:00.511 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 196 M 2023-12-26T21:53:00.511 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:53:00.884 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): libcephsqlite-16.2.14-383.g2c2e7b98.el8. 950 kB/s | 174 kB 00:00 2023-12-26T21:53:00.918 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): jq-1.6-8.el8.x86_64.rpm 939 kB/s | 203 kB 00:00 2023-12-26T21:53:01.018 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): socat-1.7.4.1-1.el8.x86_64.rpm 3.2 MB/s | 323 kB 00:00 2023-12-26T21:53:01.093 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 940 kB/s | 71 kB 00:00 2023-12-26T21:53:01.110 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): oniguruma-6.8.2-2.el8.x86_64.rpm 830 kB/s | 187 kB 00:00 2023-12-26T21:53:01.871 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/6 2023-12-26T21:53:02.122 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/6 2023-12-26T21:53:02.394 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/6 2023-12-26T21:53:02.394 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/6 2023-12-26T21:53:02.394 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/6 2023-12-26T21:53:02.394 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-26T21:53:02.394 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-26T21:53:02.394 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: jq-1.6-8.el8.x86_64 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-26T21:53:03.157 INFO:teuthology.orchestra.run.smithi076.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-26T21:53:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-26T21:53:03.158 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:03.158 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:03.357 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph 2023-12-26T21:53:03.385 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): ceph-test-16.2.14-383.g2c2e7b98.el8.x86_ 19 MB/s | 52 MB 00:02 2023-12-26T21:53:03.387 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:03.388 INFO:teuthology.orchestra.run.smithi059.stdout:Total 18 MB/s | 53 MB 00:02 2023-12-26T21:53:03.433 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:53:03.455 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:53:03.455 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:53:03.777 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:53:03.781 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:53:04.217 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:00:55 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:04.308 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:53:04.528 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-12-26T21:53:04.692 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-12-26T21:53:04.830 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-26T21:53:04.907 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-12-26T21:53:05.108 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-12-26T21:53:05.288 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/6 2023-12-26T21:53:05.901 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:05.904 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:05.904 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: ceph x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 6.4 k 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 2.4 M 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 1.6 M 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 245 k 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 5.0 M 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 18 M 2023-12-26T21:53:05.905 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-26T21:53:05.906 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-26T21:53:05.907 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-26T21:53:05.908 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:05.908 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:53:05.908 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:05.908 INFO:teuthology.orchestra.run.smithi076.stdout:Install 32 Packages 2023-12-26T21:53:05.908 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:05.908 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 31 M 2023-12-26T21:53:05.909 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 101 M 2023-12-26T21:53:05.909 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:53:06.662 INFO:teuthology.orchestra.run.smithi076.stdout:(1/32): ceph-16.2.14-383.g2c2e7b98.el8.x86_64.r 33 kB/s | 6.4 kB 00:00 2023-12-26T21:53:06.821 INFO:teuthology.orchestra.run.smithi076.stdout:(2/32): ceph-mds-16.2.14-383.g2c2e7b98.el8.x86_ 6.9 MB/s | 2.4 MB 00:00 2023-12-26T21:53:06.885 INFO:teuthology.orchestra.run.smithi076.stdout:(3/32): ceph-mgr-16.2.14-383.g2c2e7b98.el8.x86_ 3.9 MB/s | 1.6 MB 00:00 2023-12-26T21:53:07.121 INFO:teuthology.orchestra.run.smithi076.stdout:(4/32): ceph-mgr-modules-core-16.2.14-383.g2c2e 1.0 MB/s | 245 kB 00:00 2023-12-26T21:53:07.322 INFO:teuthology.orchestra.run.smithi076.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 786 kB/s | 157 kB 00:00 2023-12-26T21:53:07.380 INFO:teuthology.orchestra.run.smithi076.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 667 kB/s | 39 kB 00:00 2023-12-26T21:53:07.447 INFO:teuthology.orchestra.run.smithi076.stdout:(7/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 1.5 MB/s | 103 kB 00:00 2023-12-26T21:53:07.506 INFO:teuthology.orchestra.run.smithi076.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 978 kB/s | 57 kB 00:00 2023-12-26T21:53:07.748 INFO:teuthology.orchestra.run.smithi076.stdout:(9/32): ceph-osd-16.2.14-383.g2c2e7b98.el8.x86_ 20 MB/s | 18 MB 00:00 2023-12-26T21:53:07.773 INFO:teuthology.orchestra.run.smithi076.stdout:(10/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.7 MB/s | 457 kB 00:00 2023-12-26T21:53:08.032 INFO:teuthology.orchestra.run.smithi076.stdout:(11/32): ceph-mon-16.2.14-383.g2c2e7b98.el8.x86 3.7 MB/s | 5.0 MB 00:01 2023-12-26T21:53:08.099 INFO:teuthology.orchestra.run.smithi076.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 710 kB/s | 249 kB 00:00 2023-12-26T21:53:08.241 INFO:teuthology.orchestra.run.smithi076.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.3 MB/s | 641 kB 00:00 2023-12-26T21:53:08.285 INFO:teuthology.orchestra.run.smithi076.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 432 kB/s | 109 kB 00:00 2023-12-26T21:53:08.400 INFO:teuthology.orchestra.run.smithi076.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 145 kB/s | 44 kB 00:00 2023-12-26T21:53:08.650 INFO:teuthology.orchestra.run.smithi076.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 453 kB/s | 185 kB 00:00 2023-12-26T21:53:08.675 INFO:teuthology.orchestra.run.smithi076.stdout:(17/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.4 MB/s | 384 kB 00:00 2023-12-26T21:53:08.709 INFO:teuthology.orchestra.run.smithi076.stdout:(18/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 407 kB/s | 173 kB 00:00 2023-12-26T21:53:08.751 INFO:teuthology.orchestra.run.smithi076.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 110 kB/s | 11 kB 00:00 2023-12-26T21:53:08.784 INFO:teuthology.orchestra.run.smithi076.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 161 kB/s | 18 kB 00:00 2023-12-26T21:53:08.818 INFO:teuthology.orchestra.run.smithi076.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 447 kB/s | 49 kB 00:00 2023-12-26T21:53:08.852 INFO:teuthology.orchestra.run.smithi076.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 584 kB/s | 59 kB 00:00 2023-12-26T21:53:08.902 INFO:teuthology.orchestra.run.smithi076.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 186 kB/s | 16 kB 00:00 2023-12-26T21:53:08.936 INFO:teuthology.orchestra.run.smithi076.stdout:(24/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.8 MB/s | 283 kB 00:00 2023-12-26T21:53:08.961 INFO:teuthology.orchestra.run.smithi076.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 171 kB/s | 19 kB 00:00 2023-12-26T21:53:09.012 INFO:teuthology.orchestra.run.smithi076.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 220 kB/s | 24 kB 00:00 2023-12-26T21:53:09.037 INFO:teuthology.orchestra.run.smithi076.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 290 kB/s | 29 kB 00:00 2023-12-26T21:53:09.062 INFO:teuthology.orchestra.run.smithi076.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 273 kB/s | 27 kB 00:00 2023-12-26T21:53:09.112 INFO:teuthology.orchestra.run.smithi076.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.5 MB/s | 255 kB 00:00 2023-12-26T21:53:09.146 INFO:teuthology.orchestra.run.smithi076.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.0 MB/s | 85 kB 00:00 2023-12-26T21:53:09.188 INFO:teuthology.orchestra.run.smithi076.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2023-12-26T21:53:09.213 INFO:teuthology.orchestra.run.smithi076.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.4 MB/s | 251 kB 00:00 2023-12-26T21:53:09.217 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:09.217 INFO:teuthology.orchestra.run.smithi076.stdout:Total 9.5 MB/s | 31 MB 00:03 2023-12-26T21:53:09.300 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:53:09.339 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:53:09.339 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:53:09.664 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:53:09.667 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:53:10.212 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/6 2023-12-26T21:53:10.453 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/6 2023-12-26T21:53:10.766 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 6/6 2023-12-26T21:53:10.767 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/6 2023-12-26T21:53:10.767 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/6 2023-12-26T21:53:10.767 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-12-26T21:53:10.767 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-12-26T21:53:10.767 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-12-26T21:53:10.796 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:53:10.994 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-26T21:53:11.222 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-26T21:53:11.456 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-26T21:53:11.576 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-12-26T21:53:11.576 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-test-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout: jq-1.6-8.el8.x86_64 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout: libcephsqlite-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout: oniguruma-6.8.2-2.el8.x86_64 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout: socat-1.7.4.1-1.el8.x86_64 2023-12-26T21:53:11.577 INFO:teuthology.orchestra.run.smithi059.stdout: xmlstarlet-1.6.1-20.el8.x86_64 2023-12-26T21:53:11.578 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:11.578 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:11.670 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-26T21:53:11.839 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph 2023-12-26T21:53:11.877 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-26T21:53:12.056 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-26T21:53:12.234 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-26T21:53:12.420 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-26T21:53:12.661 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-26T21:53:12.732 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:00:58 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:12.873 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-26T21:53:13.095 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-26T21:53:13.343 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-26T21:53:13.722 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-26T21:53:13.983 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-26T21:53:14.172 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-26T21:53:14.443 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-26T21:53:14.554 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout: ceph x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 6.4 k 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 2.4 M 2023-12-26T21:53:14.558 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 1.6 M 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 245 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 5.0 M 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 18 M 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-12-26T21:53:14.559 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-12-26T21:53:14.560 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout:Install 32 Packages 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 31 M 2023-12-26T21:53:14.561 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 101 M 2023-12-26T21:53:14.562 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:53:14.673 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-26T21:53:14.964 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-26T21:53:14.977 INFO:teuthology.orchestra.run.smithi059.stdout:(1/32): ceph-16.2.14-383.g2c2e7b98.el8.x86_64.r 45 kB/s | 6.4 kB 00:00 2023-12-26T21:53:15.136 INFO:teuthology.orchestra.run.smithi059.stdout:(2/32): ceph-mgr-16.2.14-383.g2c2e7b98.el8.x86_ 5.4 MB/s | 1.6 MB 00:00 2023-12-26T21:53:15.195 INFO:teuthology.orchestra.run.smithi059.stdout:(3/32): ceph-mds-16.2.14-383.g2c2e7b98.el8.x86_ 6.8 MB/s | 2.4 MB 00:00 2023-12-26T21:53:15.227 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-26T21:53:15.287 INFO:teuthology.orchestra.run.smithi059.stdout:(4/32): ceph-mgr-modules-core-16.2.14-383.g2c2e 2.6 MB/s | 245 kB 00:00 2023-12-26T21:53:15.433 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-26T21:53:15.504 INFO:teuthology.orchestra.run.smithi059.stdout:(5/32): python3-mako-1.0.6-14.el8.noarch.rpm 725 kB/s | 157 kB 00:00 2023-12-26T21:53:15.562 INFO:teuthology.orchestra.run.smithi059.stdout:(6/32): python3-markupsafe-0.23-19.el8.x86_64.r 665 kB/s | 39 kB 00:00 2023-12-26T21:53:15.644 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-26T21:53:15.646 INFO:teuthology.orchestra.run.smithi059.stdout:(7/32): ceph-mon-16.2.14-383.g2c2e7b98.el8.x86_ 7.5 MB/s | 5.0 MB 00:00 2023-12-26T21:53:15.671 INFO:teuthology.orchestra.run.smithi059.stdout:(8/32): python3-pyOpenSSL-19.0.0-1.el8.noarch.r 954 kB/s | 103 kB 00:00 2023-12-26T21:53:15.797 INFO:teuthology.orchestra.run.smithi059.stdout:(9/32): python3-werkzeug-0.12.2-4.el8.noarch.rp 3.6 MB/s | 457 kB 00:00 2023-12-26T21:53:15.814 INFO:teuthology.orchestra.run.smithi059.stdout:(10/32): python3-pytz-2017.2-11.el8.noarch.rpm 341 kB/s | 57 kB 00:00 2023-12-26T21:53:15.879 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-26T21:53:16.100 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-26T21:53:16.164 INFO:teuthology.orchestra.run.smithi059.stdout:(11/32): ceph-osd-16.2.14-383.g2c2e7b98.el8.x86 18 MB/s | 18 MB 00:01 2023-12-26T21:53:16.220 INFO:teuthology.orchestra.run.smithi059.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 588 kB/s | 249 kB 00:00 2023-12-26T21:53:16.289 INFO:teuthology.orchestra.run.smithi059.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 1.3 MB/s | 641 kB 00:00 2023-12-26T21:53:16.307 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-26T21:53:16.340 INFO:teuthology.orchestra.run.smithi059.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 365 kB/s | 44 kB 00:00 2023-12-26T21:53:16.390 INFO:teuthology.orchestra.run.smithi059.stdout:(15/32): python3-pycparser-2.14-14.el8.noarch.r 484 kB/s | 109 kB 00:00 2023-12-26T21:53:16.473 INFO:teuthology.orchestra.run.smithi059.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.3 MB/s | 173 kB 00:00 2023-12-26T21:53:16.507 INFO:teuthology.orchestra.run.smithi059.stdout:(17/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 852 kB/s | 185 kB 00:00 2023-12-26T21:53:16.533 INFO:teuthology.orchestra.run.smithi059.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 188 kB/s | 11 kB 00:00 2023-12-26T21:53:16.558 INFO:teuthology.orchestra.run.smithi059.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 350 kB/s | 18 kB 00:00 2023-12-26T21:53:16.583 INFO:teuthology.orchestra.run.smithi059.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 970 kB/s | 49 kB 00:00 2023-12-26T21:53:16.608 INFO:teuthology.orchestra.run.smithi059.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.1 MB/s | 59 kB 00:00 2023-12-26T21:53:16.624 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-26T21:53:16.633 INFO:teuthology.orchestra.run.smithi059.stdout:(22/32): python3-cherrypy-18.4.0-1.el8.noarch.r 1.5 MB/s | 384 kB 00:00 2023-12-26T21:53:16.659 INFO:teuthology.orchestra.run.smithi059.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 310 kB/s | 16 kB 00:00 2023-12-26T21:53:16.722 INFO:teuthology.orchestra.run.smithi059.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 213 kB/s | 19 kB 00:00 2023-12-26T21:53:16.751 INFO:teuthology.orchestra.run.smithi059.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 260 kB/s | 24 kB 00:00 2023-12-26T21:53:16.776 INFO:teuthology.orchestra.run.smithi059.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 539 kB/s | 29 kB 00:00 2023-12-26T21:53:16.802 INFO:teuthology.orchestra.run.smithi059.stdout:(27/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2023-12-26T21:53:16.827 INFO:teuthology.orchestra.run.smithi059.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2023-12-26T21:53:16.852 INFO:teuthology.orchestra.run.smithi059.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.3 MB/s | 255 kB 00:00 2023-12-26T21:53:16.878 INFO:teuthology.orchestra.run.smithi059.stdout:(30/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 3.2 MB/s | 251 kB 00:00 2023-12-26T21:53:16.903 INFO:teuthology.orchestra.run.smithi059.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 458 kB/s | 23 kB 00:00 2023-12-26T21:53:16.920 INFO:teuthology.orchestra.run.smithi059.stdout:(32/32): python3-webtest-2.0.33-1.el8.noarch.rp 929 kB/s | 85 kB 00:00 2023-12-26T21:53:16.921 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:16.921 INFO:teuthology.orchestra.run.smithi059.stdout:Total 13 MB/s | 31 MB 00:02 2023-12-26T21:53:16.965 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-26T21:53:17.011 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:53:17.053 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:53:17.053 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:53:17.360 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-383.g2c2e7b98.el8. 27/32 2023-12-26T21:53:17.388 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:53:17.391 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:53:17.558 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 28/32 2023-12-26T21:53:18.615 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:53:18.855 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-12-26T21:53:19.047 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 28/32 2023-12-26T21:53:19.087 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-12-26T21:53:19.222 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 29/32 2023-12-26T21:53:19.368 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-12-26T21:53:19.602 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-12-26T21:53:19.650 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 29/32 2023-12-26T21:53:19.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-12-26T21:53:19.838 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 30/32 2023-12-26T21:53:20.050 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-12-26T21:53:20.054 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 30/32 2023-12-26T21:53:20.222 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 31/32 2023-12-26T21:53:20.239 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 31/32 2023-12-26T21:53:20.257 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-12-26T21:53:20.396 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 32/32 2023-12-26T21:53:20.468 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-12-26T21:53:20.714 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 32/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 4/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-383.g2c2e7b98.el8. 6/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-26T21:53:20.800 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-26T21:53:20.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-26T21:53:20.802 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-26T21:53:20.802 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-26T21:53:20.802 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-26T21:53:20.802 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-26T21:53:20.803 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-26T21:53:20.803 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-26T21:53:20.803 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-26T21:53:20.803 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-26T21:53:20.967 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-12-26T21:53:21.238 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-12-26T21:53:21.513 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-12-26T21:53:21.928 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-modules-core-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:21.986 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-26T21:53:21.987 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-portend-2.6-1.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-26T21:53:21.988 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:21.989 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:22.266 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-12-26T21:53:22.348 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-base 2023-12-26T21:53:22.478 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-12-26T21:53:22.941 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-12-26T21:53:23.165 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-26T21:53:23.208 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:14 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:23.574 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-12-26T21:53:23.869 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-12-26T21:53:24.066 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-12-26T21:53:24.296 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-12-26T21:53:24.588 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-12-26T21:53:24.693 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:53:24.817 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-12-26T21:53:24.918 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:24.920 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:53:24.920 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:25.023 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install cephadm 2023-12-26T21:53:25.089 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-12-26T21:53:25.414 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-12-26T21:53:25.787 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-12-26T21:53:25.874 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:16 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:26.343 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-modules-core-2:16.2.14-383.g2c2e7b98.el8. 27/32 2023-12-26T21:53:26.565 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 28/32 2023-12-26T21:53:27.546 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:27.546 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 85 k 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 85 k 2023-12-26T21:53:27.547 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 352 k 2023-12-26T21:53:27.548 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:53:27.806 INFO:teuthology.orchestra.run.smithi076.stdout:cephadm-16.2.14-383.g2c2e7b98.el8.noarch.rpm 330 kB/s | 85 kB 00:00 2023-12-26T21:53:27.806 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:27.807 INFO:teuthology.orchestra.run.smithi076.stdout:Total 328 kB/s | 85 kB 00:00 2023-12-26T21:53:27.807 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:53:27.814 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:53:27.814 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:53:27.816 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:53:27.819 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:53:27.867 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:53:28.152 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 28/32 2023-12-26T21:53:28.362 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 29/32 2023-12-26T21:53:28.414 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:28.586 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:28.811 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:28.828 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 29/32 2023-12-26T21:53:29.328 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 30/32 2023-12-26T21:53:29.494 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:29.494 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:29.494 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:53:29.494 INFO:teuthology.orchestra.run.smithi076.stdout: cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:29.495 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:29.495 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:29.562 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 30/32 2023-12-26T21:53:29.721 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-immutable-object-cache 2023-12-26T21:53:29.769 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 31/32 2023-12-26T21:53:29.788 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 31/32 2023-12-26T21:53:29.927 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 32/32 2023-12-26T21:53:30.373 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 32/32 2023-12-26T21:53:30.373 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 3/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 4/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 5/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-modules-core-2:16.2.14-383.g2c2e7b98.el8. 6/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-12-26T21:53:30.374 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-12-26T21:53:30.375 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-12-26T21:53:30.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-12-26T21:53:30.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-12-26T21:53:30.376 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-12-26T21:53:30.574 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:21 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:31.620 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-12-26T21:53:31.620 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:31.620 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:53:31.620 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:31.620 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mds-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:31.620 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-modules-core-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mon-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-osd-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-12-26T21:53:31.621 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-6.2-6.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-mako-1.0.6-14.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-portend-2.6-1.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pycparser-2.14-14.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pytz-2017.2-11.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-12-26T21:53:31.622 INFO:teuthology.orchestra.run.smithi059.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-12-26T21:53:31.623 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:31.624 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:31.816 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-base 2023-12-26T21:53:32.259 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:32.260 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 161 k 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:32.261 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:53:32.262 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:32.262 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 161 k 2023-12-26T21:53:32.262 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 470 k 2023-12-26T21:53:32.262 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:53:32.670 INFO:teuthology.orchestra.run.smithi076.stdout:ceph-immutable-object-cache-16.2.14-383.g2c2e7b 394 kB/s | 161 kB 00:00 2023-12-26T21:53:32.671 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:32.671 INFO:teuthology.orchestra.run.smithi076.stdout:Total 393 kB/s | 161 kB 00:00 2023-12-26T21:53:32.671 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:53:32.683 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:53:32.683 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:53:32.708 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:18 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:32.734 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:53:32.737 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:53:33.573 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:53:34.255 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-base-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:53:34.361 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98. 1/1 2023-12-26T21:53:34.497 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:34.498 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:53:34.499 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:34.618 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install cephadm 2023-12-26T21:53:34.827 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98. 1/1 2023-12-26T21:53:35.485 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98. 1/1 2023-12-26T21:53:35.485 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:35.485 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:53:35.485 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:35.486 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:35.486 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:35.505 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:21 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:35.692 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr 2023-12-26T21:53:36.552 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:27 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:37.297 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 85 k 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:53:37.298 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:37.299 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:53:37.299 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:37.299 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 85 k 2023-12-26T21:53:37.299 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 352 k 2023-12-26T21:53:37.299 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:53:37.460 INFO:teuthology.orchestra.run.smithi059.stdout:cephadm-16.2.14-383.g2c2e7b98.el8.noarch.rpm 531 kB/s | 85 kB 00:00 2023-12-26T21:53:37.460 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:37.461 INFO:teuthology.orchestra.run.smithi059.stdout:Total 527 kB/s | 85 kB 00:00 2023-12-26T21:53:37.461 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:53:37.468 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:53:37.468 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:53:37.470 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:53:37.474 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:53:37.515 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:53:38.024 INFO:teuthology.orchestra.run.smithi076.stdout:Package ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:53:38.058 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:38.244 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:38.253 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:38.254 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:53:38.254 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:38.359 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-dashboard 2023-12-26T21:53:38.478 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/1 2023-12-26T21:53:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:39.206 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:53:39.206 INFO:teuthology.orchestra.run.smithi059.stdout: cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:39.207 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:39.207 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:39.213 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:30 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:39.460 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-immutable-object-cache 2023-12-26T21:53:40.358 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:26 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:40.893 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:40.894 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:40.894 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:53:40.894 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:40.894 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 3.1 M 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 24 k 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 15 k 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2023-12-26T21:53:40.895 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:40.896 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 3.4 M 2023-12-26T21:53:40.896 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 22 M 2023-12-26T21:53:40.896 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:53:41.305 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): ceph-prometheus-alerts-16.2.14-383.g2c2e 87 kB/s | 15 kB 00:00 2023-12-26T21:53:41.364 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): ceph-grafana-dashboards-16.2.14-383.g2c2 107 kB/s | 24 kB 00:00 2023-12-26T21:53:41.531 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): ceph-mgr-dashboard-16.2.14-383.g2c2e7b98 7.9 MB/s | 3.1 MB 00:00 2023-12-26T21:53:41.556 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 170 kB/s | 43 kB 00:00 2023-12-26T21:53:41.673 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 109 kB/s | 34 kB 00:00 2023-12-26T21:53:42.177 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 161 k 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:42.178 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:53:42.179 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:42.179 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:53:42.179 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:42.179 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 161 k 2023-12-26T21:53:42.179 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 470 k 2023-12-26T21:53:42.180 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:53:42.240 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 277 kB/s | 196 kB 00:00 2023-12-26T21:53:42.243 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:42.244 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.5 MB/s | 3.4 MB 00:01 2023-12-26T21:53:42.255 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:53:42.266 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:53:42.266 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:53:42.399 INFO:teuthology.orchestra.run.smithi059.stdout:ceph-immutable-object-cache-16.2.14-383.g2c2e7b 735 kB/s | 161 kB 00:00 2023-12-26T21:53:42.400 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:42.400 INFO:teuthology.orchestra.run.smithi059.stdout:Total 731 kB/s | 161 kB 00:00 2023-12-26T21:53:42.400 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:53:42.412 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:53:42.412 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:53:42.425 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:53:42.429 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:53:42.463 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:53:42.466 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:53:42.585 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:53:42.711 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:53:42.795 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98. 1/1 2023-12-26T21:53:42.949 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-26T21:53:43.186 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-26T21:53:43.238 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98. 1/1 2023-12-26T21:53:43.371 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-26T21:53:43.483 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-383.g2c2e7b98.el8.n 4/6 2023-12-26T21:53:43.949 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98. 1/1 2023-12-26T21:53:43.949 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:43.949 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:53:43.949 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-immutable-object-cache-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:53:43.950 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:43.950 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:44.184 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr 2023-12-26T21:53:44.567 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-383.g2c2e7b98.el8. 5/6 2023-12-26T21:53:45.076 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:31 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:45.146 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarc 6/6 2023-12-26T21:53:45.425 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarc 6/6 2023-12-26T21:53:45.426 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-383.g2c2e7b98.el8. 1/6 2023-12-26T21:53:45.426 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarc 2/6 2023-12-26T21:53:45.426 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-383.g2c2e7b98.el8.n 3/6 2023-12-26T21:53:45.426 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-26T21:53:45.426 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-26T21:53:46.147 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-grafana-dashboards-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-prometheus-alerts-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:46.148 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:46.264 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-26T21:53:46.635 INFO:teuthology.orchestra.run.smithi059.stdout:Package ceph-mgr-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:53:46.880 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:46.881 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:53:46.881 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:47.025 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-dashboard 2023-12-26T21:53:47.129 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:38 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:53:47.911 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:33 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:48.804 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 8.6 M 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-26T21:53:48.805 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout:============================================================================================ 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout:Install 8 Packages 2023-12-26T21:53:48.806 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:48.807 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 37 M 2023-12-26T21:53:48.807 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 210 M 2023-12-26T21:53:48.807 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:53:49.521 INFO:teuthology.orchestra.run.smithi076.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 9.9 MB/s | 4.7 MB 00:00 2023-12-26T21:53:49.605 INFO:teuthology.orchestra.run.smithi076.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 8.7 MB/s | 4.9 MB 00:00 2023-12-26T21:53:49.664 INFO:teuthology.orchestra.run.smithi076.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 3.8 MB/s | 225 kB 00:00 2023-12-26T21:53:49.722 INFO:teuthology.orchestra.run.smithi076.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 19 MB/s | 3.7 MB 00:00 2023-12-26T21:53:49.732 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:49.733 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:49.733 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:53:49.733 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:49.733 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 3.1 M 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 24 k 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 15 k 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-12-26T21:53:49.734 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:49.735 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.4 M 2023-12-26T21:53:49.735 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 22 M 2023-12-26T21:53:49.735 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:53:50.081 INFO:teuthology.orchestra.run.smithi076.stdout:(5/8): python3-scipy-1.0.0-21.module_el8+762+77 34 MB/s | 14 MB 00:00 2023-12-26T21:53:50.156 INFO:teuthology.orchestra.run.smithi076.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.5 MB/s | 645 kB 00:00 2023-12-26T21:53:50.337 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): ceph-grafana-dashboards-16.2.14-383.g2c2 107 kB/s | 24 kB 00:00 2023-12-26T21:53:50.390 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-prometheus-alerts-16.2.14-383.g2c2e 53 kB/s | 15 kB 00:00 2023-12-26T21:53:50.426 INFO:teuthology.orchestra.run.smithi076.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 501 kB/s | 172 kB 00:00 2023-12-26T21:53:50.571 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-jwt-1.6.1-2.el8.noarch.rpm 182 kB/s | 43 kB 00:00 2023-12-26T21:53:50.671 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): ceph-mgr-dashboard-16.2.14-383.g2c2e7b98 5.6 MB/s | 3.1 MB 00:00 2023-12-26T21:53:50.696 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 110 kB/s | 34 kB 00:00 2023-12-26T21:53:51.058 INFO:teuthology.orchestra.run.smithi076.stdout:(8/8): ceph-mgr-diskprediction-local-16.2.14-38 4.3 MB/s | 8.6 MB 00:02 2023-12-26T21:53:51.059 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:51.059 INFO:teuthology.orchestra.run.smithi076.stdout:Total 16 MB/s | 37 MB 00:02 2023-12-26T21:53:51.108 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 365 kB/s | 196 kB 00:00 2023-12-26T21:53:51.110 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:51.111 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.5 MB/s | 3.4 MB 00:01 2023-12-26T21:53:51.123 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:53:51.134 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:53:51.134 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:53:51.292 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:53:51.307 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:53:51.310 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:53:51.310 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:53:51.310 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:53:51.615 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:53:51.652 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:53:51.655 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:53:51.936 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-12-26T21:53:52.209 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:53:52.270 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-12-26T21:53:52.351 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-26T21:53:52.465 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-26T21:53:52.494 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-12-26T21:53:52.591 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-26T21:53:52.599 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-prometheus-alerts-2:16.2.14-383.g2c2e7b98.el8.n 4/6 2023-12-26T21:53:53.197 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-26T21:53:53.367 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-26T21:53:53.740 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-grafana-dashboards-2:16.2.14-383.g2c2e7b98.el8. 5/6 2023-12-26T21:53:53.983 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-26T21:53:54.011 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarc 6/6 2023-12-26T21:53:54.125 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-26T21:53:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarc 6/6 2023-12-26T21:53:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-grafana-dashboards-2:16.2.14-383.g2c2e7b98.el8. 1/6 2023-12-26T21:53:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarc 2/6 2023-12-26T21:53:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-prometheus-alerts-2:16.2.14-383.g2c2e7b98.el8.n 3/6 2023-12-26T21:53:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-12-26T21:53:54.341 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-12-26T21:53:54.849 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-26T21:53:55.079 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-12-26T21:53:55.079 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:55.079 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-grafana-dashboards-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-dashboard-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-prometheus-alerts-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: python3-routes-2.4.1-12.el8.noarch 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:55.080 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:53:55.187 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-26T21:53:55.201 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-diskprediction-local 2023-12-26T21:53:55.408 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-26T21:53:56.092 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:42 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:53:57.185 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-26T21:53:57.904 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:53:57.909 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2023-12-26T21:53:57.909 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:53:57.909 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2023-12-26T21:53:57.909 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:53:57.909 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 8.6 M 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout:============================================================================================ 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout:Install 8 Packages 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:53:57.910 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 37 M 2023-12-26T21:53:57.911 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 210 M 2023-12-26T21:53:57.911 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:53:58.458 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-26T21:53:58.641 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b9 8/8 2023-12-26T21:53:58.748 INFO:teuthology.orchestra.run.smithi059.stdout:(1/8): openblas-0.3.15-4.el8.x86_64.rpm 10 MB/s | 4.7 MB 00:00 2023-12-26T21:53:58.821 INFO:teuthology.orchestra.run.smithi059.stdout:(2/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 9.0 MB/s | 4.9 MB 00:00 2023-12-26T21:53:58.936 INFO:teuthology.orchestra.run.smithi059.stdout:(3/8): ceph-mgr-diskprediction-local-16.2.14-38 13 MB/s | 8.6 MB 00:00 2023-12-26T21:53:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b9 8/8 2023-12-26T21:53:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b9 1/8 2023-12-26T21:53:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-26T21:53:58.953 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-26T21:53:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-26T21:53:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-26T21:53:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-26T21:53:58.954 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-26T21:53:58.993 INFO:teuthology.orchestra.run.smithi059.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.3 MB/s | 225 kB 00:00 2023-12-26T21:53:59.062 INFO:teuthology.orchestra.run.smithi059.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 12 MB/s | 3.7 MB 00:00 2023-12-26T21:53:59.279 INFO:teuthology.orchestra.run.smithi059.stdout:(6/8): libquadmath-8.5.0-21.el8.x86_64.rpm 795 kB/s | 172 kB 00:00 2023-12-26T21:53:59.320 INFO:teuthology.orchestra.run.smithi059.stdout:(7/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.9 MB/s | 645 kB 00:00 2023-12-26T21:53:59.646 INFO:teuthology.orchestra.run.smithi059.stdout:(8/8): python3-scipy-1.0.0-21.module_el8+762+77 20 MB/s | 14 MB 00:00 2023-12-26T21:53:59.646 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:53:59.647 INFO:teuthology.orchestra.run.smithi059.stdout:Total 21 MB/s | 37 MB 00:01 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-26T21:53:59.675 INFO:teuthology.orchestra.run.smithi076.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-26T21:53:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-26T21:53:59.676 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:53:59.676 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:53:59.798 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-rook 2023-12-26T21:53:59.886 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:53:59.905 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:53:59.905 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:54:00.248 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:54:00.253 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:54:00.652 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:01:51 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:00.871 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:54:01.031 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-26T21:54:01.176 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-12-26T21:54:01.318 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-26T21:54:01.975 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-12-26T21:54:02.164 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-12-26T21:54:02.330 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:02.332 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:02.332 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:54:02.332 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:02.332 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:02.332 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 27 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-26T21:54:02.333 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout:Install 13 Packages 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 1.9 M 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 24 M 2023-12-26T21:54:02.334 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:02.816 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-12-26T21:54:03.009 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-26T21:54:04.018 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-12-26T21:54:04.406 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-12-26T21:54:04.574 INFO:teuthology.orchestra.run.smithi076.stdout:(1/13): ceph-mgr-rook-16.2.14-383.g2c2e7b98.el8 196 kB/s | 27 kB 00:00 2023-12-26T21:54:04.599 INFO:teuthology.orchestra.run.smithi076.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 161 kB/s | 27 kB 00:00 2023-12-26T21:54:04.624 INFO:teuthology.orchestra.run.smithi076.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 105 kB/s | 20 kB 00:00 2023-12-26T21:54:04.659 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-26T21:54:04.766 INFO:teuthology.orchestra.run.smithi076.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 658 kB/s | 126 kB 00:00 2023-12-26T21:54:04.791 INFO:teuthology.orchestra.run.smithi076.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 659 kB/s | 110 kB 00:00 2023-12-26T21:54:04.883 INFO:teuthology.orchestra.run.smithi076.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 598 kB/s | 155 kB 00:00 2023-12-26T21:54:04.909 INFO:teuthology.orchestra.run.smithi076.stdout:(7/13): python3-cachetools-3.1.1-4.el8.noarch.r 282 kB/s | 33 kB 00:00 2023-12-26T21:54:04.984 INFO:teuthology.orchestra.run.smithi076.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 163 kB/s | 16 kB 00:00 2023-12-26T21:54:05.009 INFO:teuthology.orchestra.run.smithi076.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 175 kB/s | 43 kB 00:00 2023-12-26T21:54:05.035 INFO:teuthology.orchestra.run.smithi076.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 648 kB/s | 81 kB 00:00 2023-12-26T21:54:05.093 INFO:teuthology.orchestra.run.smithi076.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2023-12-26T21:54:05.144 INFO:teuthology.orchestra.run.smithi076.stdout:(12/13): python3-rsa-4.9-2.el8.noarch.rpm 480 kB/s | 64 kB 00:00 2023-12-26T21:54:05.244 INFO:teuthology.orchestra.run.smithi076.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 4.5 MB/s | 1.2 MB 00:00 2023-12-26T21:54:05.244 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:05.244 INFO:teuthology.orchestra.run.smithi076.stdout:Total 672 kB/s | 1.9 MB 00:02 2023-12-26T21:54:05.300 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:05.316 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:05.316 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:05.403 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:05.406 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:06.014 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:54:06.260 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-26T21:54:06.441 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-26T21:54:06.533 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-12-26T21:54:06.651 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-26T21:54:06.897 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-26T21:54:07.097 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-26T21:54:07.337 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-26T21:54:07.646 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-26T21:54:07.878 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-26T21:54:07.927 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2023-12-26T21:54:08.122 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b9 8/8 2023-12-26T21:54:08.472 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b9 8/8 2023-12-26T21:54:08.472 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b9 1/8 2023-12-26T21:54:08.472 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-12-26T21:54:08.472 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-12-26T21:54:08.473 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-12-26T21:54:08.473 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-12-26T21:54:08.473 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2023-12-26T21:54:08.473 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-12-26T21:54:08.904 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-26T21:54:09.163 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-26T21:54:09.235 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-12-26T21:54:09.235 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:09.235 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:54:09.235 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-diskprediction-local-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:54:09.235 INFO:teuthology.orchestra.run.smithi059.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-0.3.15-4.el8.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:09.236 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:09.371 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-26T21:54:09.374 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-rook 2023-12-26T21:54:09.547 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-26T21:54:09.750 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 13/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 13/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 1/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-26T21:54:10.034 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-26T21:54:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-26T21:54:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-26T21:54:10.035 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-26T21:54:10.267 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:01:56 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-26T21:54:10.798 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:10.799 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:10.909 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-mgr-cephadm 2023-12-26T21:54:11.763 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:02 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:12.058 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:12.059 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:12.059 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 27 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-12-26T21:54:12.060 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout:Install 13 Packages 2023-12-26T21:54:12.061 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:12.062 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 1.9 M 2023-12-26T21:54:12.062 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 24 M 2023-12-26T21:54:12.062 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:54:12.893 INFO:teuthology.orchestra.run.smithi059.stdout:(1/13): python3-jsonpointer-1.10-11.el8.noarch. 140 kB/s | 20 kB 00:00 2023-12-26T21:54:12.918 INFO:teuthology.orchestra.run.smithi059.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 158 kB/s | 27 kB 00:00 2023-12-26T21:54:13.035 INFO:teuthology.orchestra.run.smithi059.stdout:(3/13): ceph-mgr-rook-16.2.14-383.g2c2e7b98.el8 96 kB/s | 27 kB 00:00 2023-12-26T21:54:13.060 INFO:teuthology.orchestra.run.smithi059.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 757 kB/s | 126 kB 00:00 2023-12-26T21:54:13.086 INFO:teuthology.orchestra.run.smithi059.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 658 kB/s | 110 kB 00:00 2023-12-26T21:54:13.277 INFO:teuthology.orchestra.run.smithi059.stdout:(6/13): python3-requests-oauthlib-1.0.0-1.el8.n 196 kB/s | 43 kB 00:00 2023-12-26T21:54:13.303 INFO:teuthology.orchestra.run.smithi059.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 579 kB/s | 155 kB 00:00 2023-12-26T21:54:13.328 INFO:teuthology.orchestra.run.smithi059.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 136 kB/s | 33 kB 00:00 2023-12-26T21:54:13.454 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:13.455 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 116 k 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:13.456 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:13.457 INFO:teuthology.orchestra.run.smithi076.stdout:Install 6 Packages 2023-12-26T21:54:13.457 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:13.457 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 5.8 M 2023-12-26T21:54:13.457 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 24 M 2023-12-26T21:54:13.457 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:13.512 INFO:teuthology.orchestra.run.smithi059.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 70 kB/s | 16 kB 00:00 2023-12-26T21:54:13.545 INFO:teuthology.orchestra.run.smithi059.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 335 kB/s | 81 kB 00:00 2023-12-26T21:54:13.612 INFO:teuthology.orchestra.run.smithi059.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 918 kB/s | 61 kB 00:00 2023-12-26T21:54:13.646 INFO:teuthology.orchestra.run.smithi059.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 3.7 MB/s | 1.2 MB 00:00 2023-12-26T21:54:13.671 INFO:teuthology.orchestra.run.smithi059.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 405 kB/s | 64 kB 00:00 2023-12-26T21:54:13.674 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:13.674 INFO:teuthology.orchestra.run.smithi059.stdout:Total 1.2 MB/s | 1.9 MB 00:01 2023-12-26T21:54:13.736 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:54:13.753 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:54:13.753 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:54:13.848 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:54:13.851 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:54:14.169 INFO:teuthology.orchestra.run.smithi076.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.6 MB/s | 538 kB 00:00 2023-12-26T21:54:14.203 INFO:teuthology.orchestra.run.smithi076.stdout:(2/6): ceph-mgr-cephadm-16.2.14-383.g2c2e7b98.e 324 kB/s | 116 kB 00:00 2023-12-26T21:54:14.386 INFO:teuthology.orchestra.run.smithi076.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 8.8 MB/s | 4.8 MB 00:00 2023-12-26T21:54:14.495 INFO:teuthology.orchestra.run.smithi076.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 65 kB/s | 21 kB 00:00 2023-12-26T21:54:14.497 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:54:14.737 INFO:teuthology.orchestra.run.smithi076.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 139 kB/s | 49 kB 00:00 2023-12-26T21:54:14.752 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-12-26T21:54:14.787 INFO:teuthology.orchestra.run.smithi076.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 609 kB/s | 355 kB 00:00 2023-12-26T21:54:14.787 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:14.788 INFO:teuthology.orchestra.run.smithi076.stdout:Total 4.4 MB/s | 5.8 MB 00:01 2023-12-26T21:54:14.841 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:14.851 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:14.851 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:14.965 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:14.968 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:15.008 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-12-26T21:54:15.245 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:54:15.258 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-12-26T21:54:15.514 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-26T21:54:15.529 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-12-26T21:54:15.783 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-26T21:54:15.796 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-12-26T21:54:16.064 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-12-26T21:54:16.385 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-12-26T21:54:16.663 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-12-26T21:54:16.747 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-26T21:54:17.094 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-26T21:54:17.288 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-26T21:54:17.780 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 6/6 2023-12-26T21:54:17.798 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-12-26T21:54:18.074 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 6/6 2023-12-26T21:54:18.074 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/6 2023-12-26T21:54:18.074 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-26T21:54:18.074 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-26T21:54:18.074 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-26T21:54:18.074 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-26T21:54:18.111 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-12-26T21:54:18.340 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-12-26T21:54:18.546 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-12-26T21:54:18.769 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 13/13 2023-12-26T21:54:18.801 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-26T21:54:18.801 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:18.801 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:54:18.802 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:54:18.802 INFO:teuthology.orchestra.run.smithi076.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-26T21:54:18.802 INFO:teuthology.orchestra.run.smithi076.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-26T21:54:18.802 INFO:teuthology.orchestra.run.smithi076.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-26T21:54:18.802 INFO:teuthology.orchestra.run.smithi076.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-26T21:54:18.802 INFO:teuthology.orchestra.run.smithi076.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-26T21:54:18.803 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:18.803 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:18.947 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install ceph-fuse 2023-12-26T21:54:19.076 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 13/13 2023-12-26T21:54:19.076 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 1/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-12-26T21:54:19.077 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-12-26T21:54:19.810 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:10 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:19.907 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-12-26T21:54:19.907 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:19.907 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:54:19.907 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-rook-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-rsa-4.9-2.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:19.908 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:20.038 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-mgr-cephadm 2023-12-26T21:54:20.932 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:06 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:21.496 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:21.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:21.496 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repository Size 2023-12-26T21:54:21.496 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 911 k 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout:Installing dependencies: 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout:Install 2 Packages 2023-12-26T21:54:21.497 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:21.498 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 994 k 2023-12-26T21:54:21.498 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 2.8 M 2023-12-26T21:54:21.498 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:21.882 INFO:teuthology.orchestra.run.smithi076.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 439 kB/s | 83 kB 00:00 2023-12-26T21:54:21.974 INFO:teuthology.orchestra.run.smithi076.stdout:(2/2): ceph-fuse-16.2.14-383.g2c2e7b98.el8.x86_ 3.2 MB/s | 911 kB 00:00 2023-12-26T21:54:21.974 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:21.974 INFO:teuthology.orchestra.run.smithi076.stdout:Total 2.0 MB/s | 994 kB 00:00 2023-12-26T21:54:21.984 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:22.005 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:22.005 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:22.091 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:22.094 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:22.264 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:54:22.556 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-26T21:54:22.723 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:22.894 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/2 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm noarch 2:16.2.14-383.g2c2e7b98.el8 ceph-noarch 116 k 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:54:22.895 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:22.896 INFO:teuthology.orchestra.run.smithi059.stdout:Install 6 Packages 2023-12-26T21:54:22.896 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:22.896 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 5.8 M 2023-12-26T21:54:22.896 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 24 M 2023-12-26T21:54:22.896 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:54:23.078 INFO:teuthology.orchestra.run.smithi059.stdout:(1/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2023-12-26T21:54:23.104 INFO:teuthology.orchestra.run.smithi059.stdout:(2/6): ceph-mgr-cephadm-16.2.14-383.g2c2e7b98.e 381 kB/s | 116 kB 00:00 2023-12-26T21:54:23.128 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/2 2023-12-26T21:54:23.128 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/2 2023-12-26T21:54:23.279 INFO:teuthology.orchestra.run.smithi059.stdout:(3/6): python3-babel-2.5.1-7.el8.noarch.rpm 10 MB/s | 4.8 MB 00:00 2023-12-26T21:54:23.500 INFO:teuthology.orchestra.run.smithi059.stdout:(4/6): python3-apipkg-1.5-6.el8.noarch.rpm 50 kB/s | 21 kB 00:00 2023-12-26T21:54:23.629 INFO:teuthology.orchestra.run.smithi059.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 140 kB/s | 49 kB 00:00 2023-12-26T21:54:23.763 INFO:teuthology.orchestra.run.smithi059.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 540 kB/s | 355 kB 00:00 2023-12-26T21:54:23.764 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:23.764 INFO:teuthology.orchestra.run.smithi059.stdout:Total 5.6 MB/s | 5.8 MB 00:01 2023-12-26T21:54:23.821 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:54:23.831 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:54:23.832 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:54:23.949 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:54:23.953 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:54:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-26T21:54:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:24.596 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:54:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-26T21:54:24.596 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:24.596 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:24.811 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados-devel 2023-12-26T21:54:24.985 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:54:25.293 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-12-26T21:54:25.645 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-12-26T21:54:25.674 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:16 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:26.704 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-12-26T21:54:27.074 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-12-26T21:54:27.324 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-12-26T21:54:27.379 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 131 k 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:27.380 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:27.381 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:27.381 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:54:27.381 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:27.382 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 131 k 2023-12-26T21:54:27.382 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 432 k 2023-12-26T21:54:27.382 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:27.511 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 6/6 2023-12-26T21:54:27.678 INFO:teuthology.orchestra.run.smithi076.stdout:librados-devel-16.2.14-383.g2c2e7b98.el8.x86_64 442 kB/s | 131 kB 00:00 2023-12-26T21:54:27.678 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:27.678 INFO:teuthology.orchestra.run.smithi076.stdout:Total 440 kB/s | 131 kB 00:00 2023-12-26T21:54:27.679 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:27.684 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:27.684 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:27.734 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:27.736 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:27.846 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 6/6 2023-12-26T21:54:27.847 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 1/6 2023-12-26T21:54:27.847 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-12-26T21:54:27.847 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-12-26T21:54:27.847 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-12-26T21:54:27.847 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-12-26T21:54:27.862 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:54:28.103 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:28.377 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-12-26T21:54:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:28.607 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:54:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-mgr-cephadm-2:16.2.14-383.g2c2e7b98.el8.noarch 2023-12-26T21:54:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-apipkg-1.5-6.el8.noarch 2023-12-26T21:54:28.607 INFO:teuthology.orchestra.run.smithi059.stdout: python3-babel-2.5.1-7.el8.noarch 2023-12-26T21:54:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-execnet-1.7.1-1.el8.noarch 2023-12-26T21:54:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-jinja2-2.10.1-3.el8.noarch 2023-12-26T21:54:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: python3-remoto-1.2.1-1.el8.noarch 2023-12-26T21:54:28.608 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:28.608 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:28.754 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install ceph-fuse 2023-12-26T21:54:29.052 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:29.053 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:54:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:54:29.053 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:29.053 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:29.253 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs2 2023-12-26T21:54:29.646 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:15 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:30.105 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:21 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:31.447 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:31.447 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repository Size 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 911 k 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout:Installing dependencies: 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout:Install 2 Packages 2023-12-26T21:54:31.448 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:31.449 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 994 k 2023-12-26T21:54:31.449 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 2.8 M 2023-12-26T21:54:31.449 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:54:31.592 INFO:teuthology.orchestra.run.smithi076.stdout:Package libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:31.715 INFO:teuthology.orchestra.run.smithi059.stdout:(1/2): fuse-2.9.7-17.el8.x86_64.rpm 382 kB/s | 83 kB 00:00 2023-12-26T21:54:31.782 INFO:teuthology.orchestra.run.smithi059.stdout:(2/2): ceph-fuse-16.2.14-383.g2c2e7b98.el8.x86_ 3.1 MB/s | 911 kB 00:00 2023-12-26T21:54:31.783 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:31.783 INFO:teuthology.orchestra.run.smithi059.stdout:Total 2.9 MB/s | 994 kB 00:00 2023-12-26T21:54:31.793 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:54:31.815 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:54:31.815 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:54:31.818 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:31.819 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:31.819 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:31.907 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:54:31.910 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:54:31.917 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install libcephfs-devel 2023-12-26T21:54:32.094 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:54:32.356 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-12-26T21:54:32.572 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/2 2023-12-26T21:54:32.775 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:23 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:33.017 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2/2 2023-12-26T21:54:33.017 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/2 2023-12-26T21:54:33.756 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-12-26T21:54:33.756 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:33.757 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:54:33.757 INFO:teuthology.orchestra.run.smithi059.stdout: ceph-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-12-26T21:54:33.757 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:33.757 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:34.016 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados-devel 2023-12-26T21:54:34.477 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:34.478 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:34.478 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2023-12-26T21:54:34.478 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:34.478 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:34.478 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 25 k 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 25 k 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 107 k 2023-12-26T21:54:34.479 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:34.702 INFO:teuthology.orchestra.run.smithi076.stdout:libcephfs-devel-16.2.14-383.g2c2e7b98.el8.x86_6 114 kB/s | 25 kB 00:00 2023-12-26T21:54:34.703 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:34.703 INFO:teuthology.orchestra.run.smithi076.stdout:Total 114 kB/s | 25 kB 00:00 2023-12-26T21:54:34.703 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:34.707 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:34.707 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:34.709 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:34.712 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:34.760 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:54:34.918 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:20 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:34.944 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:35.211 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:35.864 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:35.864 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:35.864 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:54:35.864 INFO:teuthology.orchestra.run.smithi076.stdout: libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:54:35.865 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:35.865 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:36.015 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librados2 2023-12-26T21:54:36.737 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:36.737 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:36.737 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 131 k 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:54:36.738 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:36.739 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 131 k 2023-12-26T21:54:36.739 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 432 k 2023-12-26T21:54:36.739 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:54:36.868 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:27 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:37.077 INFO:teuthology.orchestra.run.smithi059.stdout:librados-devel-16.2.14-383.g2c2e7b98.el8.x86_64 387 kB/s | 131 kB 00:00 2023-12-26T21:54:37.078 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:37.078 INFO:teuthology.orchestra.run.smithi059.stdout:Total 386 kB/s | 131 kB 00:00 2023-12-26T21:54:37.078 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:54:37.085 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:54:37.085 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:54:37.140 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:54:37.144 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:54:37.256 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:54:37.509 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:37.836 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:38.334 INFO:teuthology.orchestra.run.smithi076.stdout:Package librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:38.561 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:38.562 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:38.562 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:38.573 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:38.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:38.574 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:54:38.574 INFO:teuthology.orchestra.run.smithi059.stdout: librados-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:54:38.574 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:38.574 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:38.665 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install librbd1 2023-12-26T21:54:38.779 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs2 2023-12-26T21:54:39.518 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:30 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:39.657 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:25 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:40.961 INFO:teuthology.orchestra.run.smithi076.stdout:Package librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:41.189 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:41.190 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:41.190 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:41.225 INFO:teuthology.orchestra.run.smithi059.stdout:Package libcephfs2-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:41.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rados 2023-12-26T21:54:41.471 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:41.472 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:54:41.473 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:41.882 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install libcephfs-devel 2023-12-26T21:54:42.144 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:33 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:42.769 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:28 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:43.608 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:43.834 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:43.835 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:43.836 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:43.941 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rgw 2023-12-26T21:54:44.620 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 25 k 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:54:44.621 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:44.622 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 25 k 2023-12-26T21:54:44.622 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 107 k 2023-12-26T21:54:44.622 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:54:44.795 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:35 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:44.866 INFO:teuthology.orchestra.run.smithi059.stdout:libcephfs-devel-16.2.14-383.g2c2e7b98.el8.x86_6 104 kB/s | 25 kB 00:00 2023-12-26T21:54:44.867 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:44.867 INFO:teuthology.orchestra.run.smithi059.stdout:Total 104 kB/s | 25 kB 00:00 2023-12-26T21:54:44.867 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:54:44.871 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:54:44.871 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:54:44.873 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:54:44.877 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:54:44.935 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:54:45.133 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:45.400 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:46.098 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:46.098 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:46.099 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:54:46.099 INFO:teuthology.orchestra.run.smithi059.stdout: libcephfs-devel-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:54:46.099 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:54:46.099 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:46.220 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librados2 2023-12-26T21:54:46.243 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:46.470 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:46.471 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:46.471 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:46.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-cephfs 2023-12-26T21:54:47.112 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:33 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:47.428 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:38 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:48.664 INFO:teuthology.orchestra.run.smithi059.stdout:Package librados2-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:48.869 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:48.907 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:48.909 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:54:48.909 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:49.014 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install librbd1 2023-12-26T21:54:49.099 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:49.100 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:49.100 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:49.235 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install python3-rbd 2023-12-26T21:54:49.908 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:35 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:50.084 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:41 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:51.492 INFO:teuthology.orchestra.run.smithi059.stdout:Package librbd1-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:51.553 INFO:teuthology.orchestra.run.smithi076.stdout:Package python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:51.741 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:51.742 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:54:51.742 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:51.779 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:51.780 INFO:teuthology.orchestra.run.smithi076.stdout:Nothing to do. 2023-12-26T21:54:51.780 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:51.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-fuse 2023-12-26T21:54:51.953 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rados 2023-12-26T21:54:52.754 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:43 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:52.841 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:38 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:54.397 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rados-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:54.441 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:54.441 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:54.441 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2023-12-26T21:54:54.441 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:54.441 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:54.441 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 88 k 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 88 k 2023-12-26T21:54:54.442 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 240 k 2023-12-26T21:54:54.443 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:54.640 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:54.641 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:54:54.641 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:54.717 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-fuse-16.2.14-383.g2c2e7b98.el8.x86_64.rpm 322 kB/s | 88 kB 00:00 2023-12-26T21:54:54.718 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:54.718 INFO:teuthology.orchestra.run.smithi076.stdout:Total 320 kB/s | 88 kB 00:00 2023-12-26T21:54:54.718 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:54.728 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:54.728 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:54.750 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rgw 2023-12-26T21:54:54.777 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:54.780 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:54.885 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:54:55.116 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:55.404 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:55.636 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:41 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:56.080 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:54:56.080 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:56.080 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:54:56.081 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:54:56.081 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:56.081 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:54:56.290 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-mirror 2023-12-26T21:54:57.152 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:48 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:54:57.200 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rgw-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:54:57.444 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:54:57.445 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:54:57.446 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:54:57.857 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-cephfs 2023-12-26T21:54:58.756 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:44 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:54:58.829 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: Package Arch Version Repo Size 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 3.7 M 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:54:58.830 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:54:58.831 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 3.7 M 2023-12-26T21:54:58.831 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 15 M 2023-12-26T21:54:58.831 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:54:59.354 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-mirror-16.2.14-383.g2c2e7b98.el8.x86_64.rpm 7.0 MB/s | 3.7 MB 00:00 2023-12-26T21:54:59.354 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:54:59.355 INFO:teuthology.orchestra.run.smithi076.stdout:Total 7.0 MB/s | 3.7 MB 00:00 2023-12-26T21:54:59.355 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:54:59.366 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:54:59.366 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:54:59.430 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:54:59.433 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:54:59.874 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:55:00.075 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:00.318 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-cephfs-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:55:00.485 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:00.562 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:55:00.563 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:55:00.564 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:55:00.671 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install python3-rbd 2023-12-26T21:55:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:55:01.181 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:55:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:55:01.181 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:55:01.181 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:55:01.410 DEBUG:teuthology.orchestra.run.smithi076:> sudo yum -y install rbd-nbd 2023-12-26T21:55:01.563 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:47 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:55:02.272 INFO:teuthology.orchestra.run.smithi076.stdout:Last metadata expiration check: 0:02:53 ago on Tue 26 Dec 2023 09:52:09 PM UTC. 2023-12-26T21:55:03.124 INFO:teuthology.orchestra.run.smithi059.stdout:Package python3-rbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 is already installed. 2023-12-26T21:55:03.367 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:55:03.369 INFO:teuthology.orchestra.run.smithi059.stdout:Nothing to do. 2023-12-26T21:55:03.369 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:55:03.478 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-fuse 2023-12-26T21:55:03.960 INFO:teuthology.orchestra.run.smithi076.stdout:Dependencies resolved. 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: Package Architecture Version Repository Size 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout:Installing: 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 175 k 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction Summary 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout:================================================================================ 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout:Install 1 Package 2023-12-26T21:55:03.961 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:55:03.962 INFO:teuthology.orchestra.run.smithi076.stdout:Total download size: 175 k 2023-12-26T21:55:03.962 INFO:teuthology.orchestra.run.smithi076.stdout:Installed size: 523 k 2023-12-26T21:55:03.962 INFO:teuthology.orchestra.run.smithi076.stdout:Downloading Packages: 2023-12-26T21:55:04.265 INFO:teuthology.orchestra.run.smithi076.stdout:rbd-nbd-16.2.14-383.g2c2e7b98.el8.x86_64.rpm 578 kB/s | 175 kB 00:00 2023-12-26T21:55:04.266 INFO:teuthology.orchestra.run.smithi076.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:55:04.266 INFO:teuthology.orchestra.run.smithi076.stdout:Total 575 kB/s | 175 kB 00:00 2023-12-26T21:55:04.267 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction check 2023-12-26T21:55:04.277 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction check succeeded. 2023-12-26T21:55:04.277 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction test 2023-12-26T21:55:04.329 INFO:teuthology.orchestra.run.smithi076.stdout:Transaction test succeeded. 2023-12-26T21:55:04.332 INFO:teuthology.orchestra.run.smithi076.stdout:Running transaction 2023-12-26T21:55:04.373 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:50 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:55:04.445 INFO:teuthology.orchestra.run.smithi076.stdout: Preparing : 1/1 2023-12-26T21:55:04.631 INFO:teuthology.orchestra.run.smithi076.stdout: Installing : rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:04.912 INFO:teuthology.orchestra.run.smithi076.stdout: Running scriptlet: rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:05.606 INFO:teuthology.orchestra.run.smithi076.stdout: Verifying : rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:05.606 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:55:05.607 INFO:teuthology.orchestra.run.smithi076.stdout:Installed: 2023-12-26T21:55:05.607 INFO:teuthology.orchestra.run.smithi076.stdout: rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:55:05.607 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:55:05.607 INFO:teuthology.orchestra.run.smithi076.stdout:Complete! 2023-12-26T21:55:05.822 DEBUG:teuthology.parallel:result is None 2023-12-26T21:55:06.166 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 88 k 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:55:06.167 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:06.168 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:55:06.168 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:06.168 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 88 k 2023-12-26T21:55:06.168 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 240 k 2023-12-26T21:55:06.168 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:55:06.423 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-fuse-16.2.14-383.g2c2e7b98.el8.x86_64.rpm 348 kB/s | 88 kB 00:00 2023-12-26T21:55:06.423 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:55:06.423 INFO:teuthology.orchestra.run.smithi059.stdout:Total 346 kB/s | 88 kB 00:00 2023-12-26T21:55:06.424 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:55:06.434 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:55:06.434 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:55:06.488 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:55:06.491 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:55:06.595 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:55:06.800 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:07.087 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:07.862 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:07.862 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:07.863 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:55:07.863 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-fuse-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:55:07.863 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:07.863 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:55:08.072 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-mirror 2023-12-26T21:55:08.968 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:02:54 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:55:10.756 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: Package Arch Version Repo Size 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 3.7 M 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:10.757 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:55:10.758 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:10.758 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 3.7 M 2023-12-26T21:55:10.758 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 15 M 2023-12-26T21:55:10.758 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:55:11.125 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-mirror-16.2.14-383.g2c2e7b98.el8.x86_64.rpm 10 MB/s | 3.7 MB 00:00 2023-12-26T21:55:11.125 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:55:11.126 INFO:teuthology.orchestra.run.smithi059.stdout:Total 10 MB/s | 3.7 MB 00:00 2023-12-26T21:55:11.126 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:55:11.139 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:55:11.140 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:55:11.208 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:55:11.211 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:55:11.673 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:55:12.039 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:12.494 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:13.223 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:13.224 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:13.224 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:55:13.224 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-mirror-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:55:13.224 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:13.224 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:55:13.431 DEBUG:teuthology.orchestra.run.smithi059:> sudo yum -y install rbd-nbd 2023-12-26T21:55:14.324 INFO:teuthology.orchestra.run.smithi059.stdout:Last metadata expiration check: 0:03:00 ago on Tue 26 Dec 2023 09:52:14 PM UTC. 2023-12-26T21:55:16.143 INFO:teuthology.orchestra.run.smithi059.stdout:Dependencies resolved. 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout: Package Architecture Version Repository Size 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:Installing: 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd x86_64 2:16.2.14-383.g2c2e7b98.el8 ceph 175 k 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:16.144 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction Summary 2023-12-26T21:55:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:================================================================================ 2023-12-26T21:55:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:Install 1 Package 2023-12-26T21:55:16.145 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:Total download size: 175 k 2023-12-26T21:55:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:Installed size: 523 k 2023-12-26T21:55:16.145 INFO:teuthology.orchestra.run.smithi059.stdout:Downloading Packages: 2023-12-26T21:55:16.435 INFO:teuthology.orchestra.run.smithi059.stdout:rbd-nbd-16.2.14-383.g2c2e7b98.el8.x86_64.rpm 603 kB/s | 175 kB 00:00 2023-12-26T21:55:16.436 INFO:teuthology.orchestra.run.smithi059.stdout:-------------------------------------------------------------------------------- 2023-12-26T21:55:16.436 INFO:teuthology.orchestra.run.smithi059.stdout:Total 600 kB/s | 175 kB 00:00 2023-12-26T21:55:16.436 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction check 2023-12-26T21:55:16.448 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction check succeeded. 2023-12-26T21:55:16.448 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction test 2023-12-26T21:55:16.502 INFO:teuthology.orchestra.run.smithi059.stdout:Transaction test succeeded. 2023-12-26T21:55:16.506 INFO:teuthology.orchestra.run.smithi059.stdout:Running transaction 2023-12-26T21:55:16.645 INFO:teuthology.orchestra.run.smithi059.stdout: Preparing : 1/1 2023-12-26T21:55:16.874 INFO:teuthology.orchestra.run.smithi059.stdout: Installing : rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:17.157 INFO:teuthology.orchestra.run.smithi059.stdout: Running scriptlet: rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:17.877 INFO:teuthology.orchestra.run.smithi059.stdout: Verifying : rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 1/1 2023-12-26T21:55:17.877 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:17.877 INFO:teuthology.orchestra.run.smithi059.stdout:Installed: 2023-12-26T21:55:17.877 INFO:teuthology.orchestra.run.smithi059.stdout: rbd-nbd-2:16.2.14-383.g2c2e7b98.el8.x86_64 2023-12-26T21:55:17.877 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:55:17.877 INFO:teuthology.orchestra.run.smithi059.stdout:Complete! 2023-12-26T21:55:18.126 DEBUG:teuthology.parallel:result is None 2023-12-26T21:55:18.127 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:55:18.127 INFO:teuthology.packaging:ref: None 2023-12-26T21:55:18.127 INFO:teuthology.packaging:tag: None 2023-12-26T21:55:18.127 INFO:teuthology.packaging:branch: pacific 2023-12-26T21:55:18.127 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:55:18.127 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-12-26T21:55:18.326 DEBUG:teuthology.orchestra.run.smithi059:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-26T21:55:18.349 INFO:teuthology.orchestra.run.smithi059.stdout:16.2.14-383.g2c2e7b98.el8 2023-12-26T21:55:18.350 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-383.g2c2e7b98.el8 2023-12-26T21:55:18.350 INFO:teuthology.task.install:The correct ceph version 16.2.14-383.g2c2e7b98 is installed. 2023-12-26T21:55:18.351 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-12-26T21:55:18.352 INFO:teuthology.packaging:ref: None 2023-12-26T21:55:18.352 INFO:teuthology.packaging:tag: None 2023-12-26T21:55:18.352 INFO:teuthology.packaging:branch: pacific 2023-12-26T21:55:18.352 INFO:teuthology.packaging:sha1: 5b5a60d2d510bfaa9213c41e32227fb008245a30 2023-12-26T21:55:18.352 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific 2023-12-26T21:55:18.633 DEBUG:teuthology.orchestra.run.smithi076:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-12-26T21:55:18.661 INFO:teuthology.orchestra.run.smithi076.stdout:16.2.14-383.g2c2e7b98.el8 2023-12-26T21:55:18.661 INFO:teuthology.packaging:The installed version of ceph is 16.2.14-383.g2c2e7b98.el8 2023-12-26T21:55:18.661 INFO:teuthology.task.install:The correct ceph version 16.2.14-383.g2c2e7b98 is installed. 2023-12-26T21:55:18.663 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-12-26T21:55:18.664 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:55:18.664 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-26T21:55:18.694 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:55:18.694 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-12-26T21:55:18.735 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-12-26T21:55:18.736 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:55:18.736 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/daemon-helper 2023-12-26T21:55:18.766 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-26T21:55:18.836 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:55:18.836 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/daemon-helper 2023-12-26T21:55:18.868 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-12-26T21:55:18.941 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-12-26T21:55:18.941 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:55:18.941 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-26T21:55:18.970 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-26T21:55:19.043 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:55:19.043 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/adjust-ulimits 2023-12-26T21:55:19.079 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-12-26T21:55:19.150 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-12-26T21:55:19.150 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:55:19.150 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd of=/usr/bin/stdin-killer 2023-12-26T21:55:19.183 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-26T21:55:19.257 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:55:19.257 DEBUG:teuthology.orchestra.run.smithi076:> sudo dd of=/usr/bin/stdin-killer 2023-12-26T21:55:19.287 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-12-26T21:55:19.356 INFO:teuthology.run_tasks:Running task print... 2023-12-26T21:55:19.581 INFO:teuthology.task.print:**** done install task... 2023-12-26T21:55:19.581 INFO:teuthology.run_tasks:Running task cephadm... 2023-12-26T21:55:19.690 INFO:tasks.cephadm:Config: {'cephadm_branch': 'pacific', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'docker.io/ceph/daemon-base:latest-pacific', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', '\\(POOL_APP_NOT_ENABLED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '5b5a60d2d510bfaa9213c41e32227fb008245a30'} 2023-12-26T21:55:19.690 INFO:tasks.cephadm:Cluster image is docker.io/ceph/daemon-base:latest-pacific 2023-12-26T21:55:19.690 INFO:tasks.cephadm:Cluster fsid is 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:19.690 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-12-26T21:55:19.690 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-12-26T21:55:19.691 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi059': '172.21.15.59', 'mon.smithi076': '172.21.15.76'} 2023-12-26T21:55:19.691 INFO:tasks.cephadm:Normalizing hostnames... 2023-12-26T21:55:19.691 DEBUG:teuthology.orchestra.run.smithi059:> sudo hostname $(hostname -s) 2023-12-26T21:55:19.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo hostname $(hostname -s) 2023-12-26T21:55:19.757 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref pacific)... 2023-12-26T21:55:19.758 DEBUG:teuthology.orchestra.run.smithi059:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-26T21:55:20.081 INFO:teuthology.orchestra.run.smithi059.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Dec 26 21:55 /home/ubuntu/cephtest/cephadm 2023-12-26T21:55:20.081 DEBUG:teuthology.orchestra.run.smithi076:> curl --silent https://raw.githubusercontent.com/ceph/ceph/pacific/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-12-26T21:55:20.246 INFO:teuthology.orchestra.run.smithi076.stdout:-rw-rw-r--. 1 ubuntu ubuntu 355729 Dec 26 21:55 /home/ubuntu/cephtest/cephadm 2023-12-26T21:55:20.246 DEBUG:teuthology.orchestra.run.smithi059:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-26T21:55:20.270 DEBUG:teuthology.orchestra.run.smithi076:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-12-26T21:55:20.303 INFO:tasks.cephadm:Pulling image docker.io/ceph/daemon-base:latest-pacific on all hosts... 2023-12-26T21:55:20.303 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2023-12-26T21:55:20.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific pull 2023-12-26T21:55:21.409 INFO:teuthology.orchestra.run.smithi076.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2023-12-26T21:55:21.562 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2023-12-26T21:55:43.744 INFO:teuthology.orchestra.run.smithi076.stdout:{ 2023-12-26T21:55:43.745 INFO:teuthology.orchestra.run.smithi076.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2023-12-26T21:55:43.745 INFO:teuthology.orchestra.run.smithi076.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2023-12-26T21:55:43.745 INFO:teuthology.orchestra.run.smithi076.stdout: "repo_digests": [ 2023-12-26T21:55:43.745 INFO:teuthology.orchestra.run.smithi076.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2023-12-26T21:55:43.745 INFO:teuthology.orchestra.run.smithi076.stdout: ] 2023-12-26T21:55:43.745 INFO:teuthology.orchestra.run.smithi076.stdout:} 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout:{ 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout: "ceph_version": "ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable)", 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout: "image_id": "46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f", 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout: "repo_digests": [ 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout: "docker.io/ceph/daemon-base@sha256:02f4aa6b553e20e1f3fae6eefcc99f66fcf867bf0eae95d362ae2121efc26684" 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout: ] 2023-12-26T21:55:45.453 INFO:teuthology.orchestra.run.smithi059.stdout:} 2023-12-26T21:55:45.475 DEBUG:teuthology.orchestra.run.smithi059:> sudo mkdir -p /etc/ceph 2023-12-26T21:55:45.509 DEBUG:teuthology.orchestra.run.smithi076:> sudo mkdir -p /etc/ceph 2023-12-26T21:55:45.549 DEBUG:teuthology.orchestra.run.smithi059:> sudo chmod 777 /etc/ceph 2023-12-26T21:55:45.579 DEBUG:teuthology.orchestra.run.smithi076:> sudo chmod 777 /etc/ceph 2023-12-26T21:55:45.622 INFO:tasks.cephadm:Writing seed config... 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-12-26T21:55:45.624 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [client] debug client = 20 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-12-26T21:55:45.625 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-12-26T21:55:45.626 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-12-26T21:55:45.627 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:55:45.627 DEBUG:teuthology.orchestra.run.smithi059:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-12-26T21:55:45.649 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 761f1ff0-a439-11ee-95a5-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-12-26T21:55:45.649 DEBUG:teuthology.orchestra.run.smithi059:mon.smithi059> sudo journalctl -f -n 0 -u ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059.service 2023-12-26T21:55:45.691 INFO:tasks.cephadm:Bootstrapping... 2023-12-26T21:55:45.692 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific -v bootstrap --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.59 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-12-26T21:55:45.719 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:-- Logs begin at Tue 2023-12-26 21:42:34 UTC. -- 2023-12-26T21:55:45.903 INFO:teuthology.orchestra.run.smithi059.stderr:-------------------------------------------------------------------------------- 2023-12-26T21:55:45.903 INFO:teuthology.orchestra.run.smithi059.stderr:cephadm ['--image', 'docker.io/ceph/daemon-base:latest-pacific', '-v', 'bootstrap', '--fsid', '761f1ff0-a439-11ee-95a5-87774f69a715', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.59', '--skip-admin-label'] 2023-12-26T21:55:45.925 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 4.6.1 2023-12-26T21:55:45.934 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying podman|docker is present... 2023-12-26T21:55:45.954 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 4.6.1 2023-12-26T21:55:45.972 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying lvm2 is present... 2023-12-26T21:55:45.972 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying time synchronization is in place... 2023-12-26T21:55:45.978 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-26T21:55:45.985 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2023-12-26T21:55:45.991 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: enabled 2023-12-26T21:55:45.996 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: active 2023-12-26T21:55:45.997 INFO:teuthology.orchestra.run.smithi059.stderr:Unit chronyd.service is enabled and running 2023-12-26T21:55:45.997 INFO:teuthology.orchestra.run.smithi059.stderr:Repeating the final host check... 2023-12-26T21:55:46.017 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 4.6.1 2023-12-26T21:55:46.029 INFO:teuthology.orchestra.run.smithi059.stderr:podman (/bin/podman) version 4.6.1 is present 2023-12-26T21:55:46.029 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl is present 2023-12-26T21:55:46.029 INFO:teuthology.orchestra.run.smithi059.stderr:lvcreate is present 2023-12-26T21:55:46.034 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-12-26T21:55:46.040 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2023-12-26T21:55:46.046 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: enabled 2023-12-26T21:55:46.052 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: active 2023-12-26T21:55:46.052 INFO:teuthology.orchestra.run.smithi059.stderr:Unit chronyd.service is enabled and running 2023-12-26T21:55:46.053 INFO:teuthology.orchestra.run.smithi059.stderr:Host looks OK 2023-12-26T21:55:46.053 INFO:teuthology.orchestra.run.smithi059.stderr:Cluster fsid: 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:46.053 INFO:teuthology.orchestra.run.smithi059.stderr:Acquiring lock 139637436496360 on /run/cephadm/761f1ff0-a439-11ee-95a5-87774f69a715.lock 2023-12-26T21:55:46.053 INFO:teuthology.orchestra.run.smithi059.stderr:Lock 139637436496360 acquired on /run/cephadm/761f1ff0-a439-11ee-95a5-87774f69a715.lock 2023-12-26T21:55:46.053 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying IP 172.21.15.59 port 3300 ... 2023-12-26T21:55:46.053 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying IP 172.21.15.59 port 6789 ... 2023-12-26T21:55:46.054 INFO:teuthology.orchestra.run.smithi059.stderr:Base mon IP(s) is [172.21.15.59:3300, 172.21.15.59:6789], mon addrv is [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2023-12-26T21:55:46.056 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: default via 172.21.15.254 dev ens1f0 proto dhcp src 172.21.15.59 metric 100 2023-12-26T21:55:46.057 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: 172.21.0.0/20 dev ens1f0 proto kernel scope link src 172.21.15.59 metric 100 2023-12-26T21:55:46.059 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-12-26T21:55:46.059 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: fe80::/64 dev ens1f0 proto kernel metric 256 pref medium 2023-12-26T21:55:46.060 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev ens1f0 proto ra metric 1024 expires 1795sec hoplimit 64 pref medium 2023-12-26T21:55:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-12-26T21:55:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-12-26T21:55:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-26T21:55:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: 4: ens1f0: mtu 1500 state UP qlen 1000 2023-12-26T21:55:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: inet6 fe80::ec4:7aff:febd:192a/64 scope link 2023-12-26T21:55:46.062 INFO:teuthology.orchestra.run.smithi059.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-12-26T21:55:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2023-12-26T21:55:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:Mon IP `172.21.15.59` is in CIDR network `172.21.0.0/20` 2023-12-26T21:55:46.063 INFO:teuthology.orchestra.run.smithi059.stderr:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2023-12-26T21:55:46.064 INFO:teuthology.orchestra.run.smithi059.stderr:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-12-26T21:55:46.065 INFO:teuthology.orchestra.run.smithi059.stderr:Pulling container image docker.io/ceph/daemon-base:latest-pacific... 2023-12-26T21:55:46.144 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Trying to pull docker.io/ceph/daemon-base:latest-pacific... 2023-12-26T21:55:52.546 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Getting image source signatures 2023-12-26T21:55:52.547 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Copying blob sha256:a997af56d24c50f31dbc8bd557a1c2adeb9ea675d7d7142f1a37cb629070c246 2023-12-26T21:55:52.547 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-12-26T21:55:52.547 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Copying config sha256:46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2023-12-26T21:55:52.611 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: Writing manifest to image destination 2023-12-26T21:55:52.768 INFO:teuthology.orchestra.run.smithi059.stderr:/bin/podman: 46cf6318c64e842f2287ebcae09f4ed8697a6751115e5245c05af1b9b2fd053f 2023-12-26T21:55:53.121 INFO:teuthology.orchestra.run.smithi059.stderr:ceph: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2023-12-26T21:55:53.481 INFO:teuthology.orchestra.run.smithi059.stderr:Ceph version: ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable) 2023-12-26T21:55:53.481 INFO:teuthology.orchestra.run.smithi059.stderr:Extracting ceph user uid/gid from container image... 2023-12-26T21:55:53.769 INFO:teuthology.orchestra.run.smithi059.stderr:stat: 167 167 2023-12-26T21:55:54.088 INFO:teuthology.orchestra.run.smithi059.stderr:Creating initial keys... 2023-12-26T21:55:54.388 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-authtool: AQDqS4tlrU5qFhAA+JVh7pM8/nH92VBKsROQ1Q== 2023-12-26T21:55:55.005 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-authtool: AQDqS4tlq6IkOxAARRNFLAiY5ndqgKiDw3GVYA== 2023-12-26T21:55:55.697 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-authtool: AQDrS4tlSHdPJBAAT0yNgIs34s8gmCtFOn0O4A== 2023-12-26T21:55:56.005 INFO:teuthology.orchestra.run.smithi059.stderr:Creating initial monmap... 2023-12-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-26T21:55:56.280 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:56.281 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-26T21:55:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:monmaptool for smithi059 [v2:172.21.15.59:3300,v1:172.21.15.59:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-12-26T21:55:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: set fsid to 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-12-26T21:55:56.639 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:55:56.639 INFO:teuthology.orchestra.run.smithi059.stderr:Creating mon... 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.896+0000 7f2aade2b700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 1 imported monmap: 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: epoch 0 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: fsid 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: last_changed 2023-12-26T21:55:56.269247+0000 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: created 2023-12-26T21:55:56.269247+0000 2023-12-26T21:55:56.932 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon.smithi059 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 0 /usr/bin/ceph-mon: set fsid to 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: RocksDB version: 6.8.1 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Compile date Jul 8 2021 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: DB SUMMARY 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 0, files: 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 2023-12-26T21:55:56.933 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.error_if_exists: 0 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.create_if_missing: 1 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.paranoid_checks: 1 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.env: 0x55b4f46061c0 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.fs: Posix File System 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.info_log: 0x55b4f68ab000 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.statistics: (nil) 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.use_fsync: 0 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_log_file_size: 0 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-26T21:55:56.934 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.allow_fallocate: 1 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.use_direct_reads: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.db_log_dir: 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_subcompactions: 1 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_background_flushes: -1 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-26T21:55:56.935 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.write_buffer_manager: 0x55b4f68b3530 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.unordered_write: 0 2023-12-26T21:55:56.936 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.row_cache: None 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.wal_filter: None 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.preserve_deletes: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.two_write_queues: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.atomic_flush: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-26T21:55:56.937 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.log_readahead_size: 0 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_background_jobs: 2 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_background_compactions: -1 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.max_open_files: -1 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-26T21:55:56.938 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Compression algorithms supported: 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kZSTD supported: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kXpressCompression supported: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kLZ4Compression supported: 1 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kBZip2Compression supported: 0 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kZlibCompression supported: 1 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: kSnappyCompression supported: 1 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.897+0000 7f2aade2b700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.939 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000001 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.merge_operator: 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_filter: None 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55b4f67c4160) 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-12-26T21:55:56.940 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: index_type: 0 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: checksum: 1 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache: 0x55b4f67fad10 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-12-26T21:55:56.941 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: format_version: 2 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-12-26T21:55:56.942 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: block_align: 0 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression: NoCompression 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.num_levels: 7 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-26T21:55:56.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-26T21:55:56.944 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-26T21:55:56.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.table_properties_collectors: 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.inplace_update_support: 0 2023-12-26T21:55:56.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.bloom_locality: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.max_successive_merges: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.ttl: 2592000 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.898+0000 7f2aade2b700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.899+0000 7f2aade2b700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.899+0000 7f2aade2b700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-12-26T21:55:56.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.899+0000 7f2aade2b700 4 rocksdb: DB pointer 0x55b4f68bf800 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.899+0000 7f2a96cb4700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.899+0000 7f2a96cb4700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:55:56.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:55:56.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-12-26T21:55:56.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.901+0000 7f2aade2b700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.901+0000 7f2aade2b700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-12-26T21:55:56.951 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph-mon: debug 2023-12-26T21:55:56.901+0000 7f2aade2b700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi059 for mon.smithi059 2023-12-26T21:55:57.244 INFO:teuthology.orchestra.run.smithi059.stderr:create mon.smithi059 on 2023-12-26T21:55:57.257 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-12-26T21:55:57.355 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-12-26T21:55:57.456 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-761f1ff0-a439-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-761f1ff0-a439-11ee-95a5-87774f69a715.target. 2023-12-26T21:55:57.456 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-761f1ff0-a439-11ee-95a5-87774f69a715.target → /etc/systemd/system/ceph-761f1ff0-a439-11ee-95a5-87774f69a715.target. 2023-12-26T21:55:57.669 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to reset failed state of unit ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059.service: Unit ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059.service not loaded. 2023-12-26T21:55:57.677 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/ceph-761f1ff0-a439-11ee-95a5-87774f69a715.target.wants/ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059.service → /etc/systemd/system/ceph-761f1ff0-a439-11ee-95a5-87774f69a715@.service. 2023-12-26T21:55:58.135 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2023-12-26T21:55:58.144 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2023-12-26T21:55:58.144 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2023-12-26T21:55:58.144 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to enable service . firewalld.service is not available 2023-12-26T21:55:58.145 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mon to start... 2023-12-26T21:55:58.145 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mon... 2023-12-26T21:55:58.166 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:58 smithi059 systemd[1]: Started Ceph mon.smithi059 for 761f1ff0-a439-11ee-95a5-87774f69a715. 2023-12-26T21:55:58.167 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.161004+0000 mon.smithi059 (mon.0) 0 : [INF] mkfs 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:58.167 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.142768+0000 mon.smithi059 (mon.0) 1 : cluster [INF] mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2023-12-26T21:55:58.594 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42b5339700 1 Processor -- start 2023-12-26T21:55:58.594 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42b5339700 1 -- start start 2023-12-26T21:55:58.594 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42b5339700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fea60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:55:58.594 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42b5339700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b00fef50 con 0x7f42b006ade0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42aeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42aeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42778/0 (socket says 172.21.15.59:42778) 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.588+0000 7f42aeffd700 1 -- 172.21.15.59:0/4015227872 learned_addr learned my addr 172.21.15.59:0/4015227872 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.589+0000 7f42aeffd700 1 -- 172.21.15.59:0/4015227872 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42b00ff090 con 0x7f42b006ade0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.589+0000 7f42aeffd700 1 --2- 172.21.15.59:0/4015227872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fea60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f42a000b5f0 tx=0x7f42a0006ee0).ready entity=mon.0 client_cookie=6f8c9d8b334565d8 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.589+0000 7f42adffb700 1 -- 172.21.15.59:0/4015227872 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42a000d600 con 0x7f42b006ade0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.589+0000 7f42adffb700 1 -- 172.21.15.59:0/4015227872 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a000d760 con 0x7f42b006ade0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.589+0000 7f42adffb700 1 -- 172.21.15.59:0/4015227872 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42a000da30 con 0x7f42b006ade0 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 -- 172.21.15.59:0/4015227872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 msgr2=0x7f42b00fea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 --2- 172.21.15.59:0/4015227872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fea60 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f42a000b5f0 tx=0x7f42a0006ee0).stop 2023-12-26T21:55:58.595 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 -- 172.21.15.59:0/4015227872 shutdown_connections 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 --2- 172.21.15.59:0/4015227872 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fea60 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 -- 172.21.15.59:0/4015227872 >> 172.21.15.59:0/4015227872 conn(0x7f42b00f6500 msgr2=0x7f42b00f8920 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 -- 172.21.15.59:0/4015227872 shutdown_connections 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.590+0000 7f42b5339700 1 -- 172.21.15.59:0/4015227872 wait complete. 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42b5339700 1 Processor -- start 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42b5339700 1 -- start start 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42b5339700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fc4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:55:58.596 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42b5339700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f42b00fa960 con 0x7f42b006ade0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42aeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fc4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42aeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fc4e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42784/0 (socket says 172.21.15.59:42784) 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42aeffd700 1 -- 172.21.15.59:0/1331011187 learned_addr learned my addr 172.21.15.59:0/1331011187 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.591+0000 7f42aeffd700 1 -- 172.21.15.59:0/1331011187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42a000b200 con 0x7f42b006ade0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42aeffd700 1 --2- 172.21.15.59:0/1331011187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fc4e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f42a0006c70 tx=0x7f42a0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42a0011b40 con 0x7f42b006ade0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f42a0011ca0 con 0x7f42b006ade0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f42b00fab00 con 0x7f42b006ade0 2023-12-26T21:55:58.597 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42a001d430 con 0x7f42b006ade0 2023-12-26T21:55:58.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42b00faf90 con 0x7f42b006ade0 2023-12-26T21:55:58.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f42a000f070 con 0x7f42b006ade0 2023-12-26T21:55:58.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.592+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f42a001dbf0 con 0x7f42b006ade0 2023-12-26T21:55:58.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.593+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42b0045bb0 con 0x7f42b006ade0 2023-12-26T21:55:58.598 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.595+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f42a0016070 con 0x7f42b006ade0 2023-12-26T21:55:58.668 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.665+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f42b0104040 con 0x7f42b006ade0 2023-12-26T21:55:58.668 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.665+0000 7f42977fe700 1 -- 172.21.15.59:0/1331011187 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f42a0036070 con 0x7f42b006ade0 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: cluster: 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: id: 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: services: 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi059 (age 0.506716s) 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mgr: no daemons active 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: data: 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-12-26T21:55:58.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: pgs: 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.665+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 msgr2=0x7f42b00fc4e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.665+0000 7f42b5339700 1 --2- 172.21.15.59:0/1331011187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fc4e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f42a0006c70 tx=0x7f42a0016000).stop 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.666+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 shutdown_connections 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.666+0000 7f42b5339700 1 --2- 172.21.15.59:0/1331011187 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42b006ade0 0x7f42b00fc4e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.666+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 >> 172.21.15.59:0/1331011187 conn(0x7f42b00f6500 msgr2=0x7f42b00f8250 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.666+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 shutdown_connections 2023-12-26T21:55:58.670 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:58.666+0000 7f42b5339700 1 -- 172.21.15.59:0/1331011187 wait complete. 2023-12-26T21:55:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:mon is available 2023-12-26T21:55:59.039 INFO:teuthology.orchestra.run.smithi059.stderr:Assimilating anything we can from ceph.conf... 2023-12-26T21:55:59.212 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.159679+0000 mon.smithi059 (mon.0) 2 : cluster [INF] mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2023-12-26T21:55:59.213 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.159913+0000 mon.smithi059 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2023-12-26T21:55:59.213 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.163200+0000 mon.smithi059 (mon.0) 4 : cluster [DBG] fsmap 2023-12-26T21:55:59.213 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.165593+0000 mon.smithi059 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-26T21:55:59.213 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: cluster 2023-12-26T21:55:58.166040+0000 mon.smithi059 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-12-26T21:55:59.213 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:55:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: audit 2023-12-26T21:55:58.666360+0000 mon.smithi059 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.59:0/1331011187' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd393a1700 1 Processor -- start 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd393a1700 1 -- start start 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd393a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd393a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340facd0 con 0x7fcd340fa3c0 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fa7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42794/0 (socket says 172.21.15.59:42794) 2023-12-26T21:55:59.451 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd32ffd700 1 -- 172.21.15.59:0/2244011214 learned_addr learned my addr 172.21.15.59:0/2244011214 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:55:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.449+0000 7fcd32ffd700 1 -- 172.21.15.59:0/2244011214 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd340fae10 con 0x7fcd340fa3c0 2023-12-26T21:55:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd32ffd700 1 --2- 172.21.15.59:0/2244011214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fa7e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fcd2400b5f0 tx=0x7fcd24006ee0).ready entity=mon.0 client_cookie=91a58dd9d173eff5 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:55:59.453 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd31ffb700 1 -- 172.21.15.59:0/2244011214 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd2400d600 con 0x7fcd340fa3c0 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd31ffb700 1 -- 172.21.15.59:0/2244011214 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd2400d760 con 0x7fcd340fa3c0 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd31ffb700 1 -- 172.21.15.59:0/2244011214 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd2400da30 con 0x7fcd340fa3c0 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 -- 172.21.15.59:0/2244011214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 msgr2=0x7fcd340fa7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 --2- 172.21.15.59:0/2244011214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fa7e0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7fcd2400b5f0 tx=0x7fcd24006ee0).stop 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 -- 172.21.15.59:0/2244011214 shutdown_connections 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 --2- 172.21.15.59:0/2244011214 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fa7e0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 -- 172.21.15.59:0/2244011214 >> 172.21.15.59:0/2244011214 conn(0x7fcd340f5e20 msgr2=0x7fcd340f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 -- 172.21.15.59:0/2244011214 shutdown_connections 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.450+0000 7fcd393a1700 1 -- 172.21.15.59:0/2244011214 wait complete. 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd393a1700 1 Processor -- start 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd393a1700 1 -- start start 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd393a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fcfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:55:59.454 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd393a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd340fb430 con 0x7fcd340fa3c0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fcfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fcfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42810/0 (socket says 172.21.15.59:42810) 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.451+0000 7fcd32ffd700 1 -- 172.21.15.59:0/111839057 learned_addr learned my addr 172.21.15.59:0/111839057 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd32ffd700 1 -- 172.21.15.59:0/111839057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd2400b200 con 0x7fcd340fa3c0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd32ffd700 1 --2- 172.21.15.59:0/111839057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fcfb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fcd24006c70 tx=0x7fcd24016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd24011b40 con 0x7fcd340fa3c0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd24011ca0 con 0x7fcd340fa3c0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd340fb5d0 con 0x7fcd340fa3c0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd2401d430 con 0x7fcd340fa3c0 2023-12-26T21:55:59.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd340fba60 con 0x7fcd340fa3c0 2023-12-26T21:55:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fcd2400f070 con 0x7fcd340fa3c0 2023-12-26T21:55:59.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.452+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcd2401dc10 con 0x7fcd340fa3c0 2023-12-26T21:55:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.453+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd34045bb0 con 0x7fcd340fa3c0 2023-12-26T21:55:59.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.455+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fcd24016070 con 0x7fcd340fa3c0 2023-12-26T21:55:59.522 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.522+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcd340008d0 con 0x7fcd340fa3c0 2023-12-26T21:55:59.526 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.525+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 7 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcd2401d650 con 0x7fcd340fa3c0 2023-12-26T21:55:59.526 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.525+0000 7fcd1b7fe700 1 -- 172.21.15.59:0/111839057 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+454 (secure 0 0 0) 0x7fcd2401c530 con 0x7fcd340fa3c0 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [global] 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: fsid = 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: log_to_journald = false 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.59:3300,v1:172.21.15.59:6789] 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-26T21:55:59.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [mgr] 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [osd] 2023-12-26T21:55:59.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 msgr2=0x7fcd340fcfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 --2- 172.21.15.59:0/111839057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fcfb0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7fcd24006c70 tx=0x7fcd24016000).stop 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 shutdown_connections 2023-12-26T21:55:59.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 --2- 172.21.15.59:0/111839057 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd340fa3c0 0x7fcd340fcfb0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:55:59.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 >> 172.21.15.59:0/111839057 conn(0x7fcd340f5e20 msgr2=0x7fcd340f7b80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:55:59.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 shutdown_connections 2023-12-26T21:55:59.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:55:59.526+0000 7fcd393a1700 1 -- 172.21.15.59:0/111839057 wait complete. 2023-12-26T21:55:59.865 INFO:teuthology.orchestra.run.smithi059.stderr:Generating new minimal ceph.conf... 2023-12-26T21:56:00.248 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.246+0000 7f9ab42c3700 1 Processor -- start 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.246+0000 7f9ab42c3700 1 -- start start 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab42c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac0fb900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab42c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0fbdf0 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab205f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac0fb900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab205f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac0fb900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42812/0 (socket says 172.21.15.59:42812) 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab205f700 1 -- 172.21.15.59:0/1883048017 learned_addr learned my addr 172.21.15.59:0/1883048017 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab205f700 1 -- 172.21.15.59:0/1883048017 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aac0fbf30 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.249 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.247+0000 7f9ab205f700 1 --2- 172.21.15.59:0/1883048017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac0fb900 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9aa800b5f0 tx=0x7f9aa8006ee0).ready entity=mon.0 client_cookie=4560f6f22c6685b7 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:00.250 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab105d700 1 -- 172.21.15.59:0/1883048017 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa800d600 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.250 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab105d700 1 -- 172.21.15.59:0/1883048017 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f9aa800dbc0 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.250 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab105d700 1 -- 172.21.15.59:0/1883048017 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa8011b90 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.250 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab42c3700 1 -- 172.21.15.59:0/1883048017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 msgr2=0x7f9aac0fb900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:00.250 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab42c3700 1 --2- 172.21.15.59:0/1883048017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac0fb900 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f9aa800b5f0 tx=0x7f9aa8006ee0).stop 2023-12-26T21:56:00.252 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab42c3700 1 -- 172.21.15.59:0/1883048017 shutdown_connections 2023-12-26T21:56:00.252 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab42c3700 1 --2- 172.21.15.59:0/1883048017 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac0fb900 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:00.252 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.248+0000 7f9ab42c3700 1 -- 172.21.15.59:0/1883048017 >> 172.21.15.59:0/1883048017 conn(0x7f9aac0f5df0 msgr2=0x7f9aac0f8210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:00.252 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.249+0000 7f9ab42c3700 1 -- 172.21.15.59:0/1883048017 shutdown_connections 2023-12-26T21:56:00.252 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.249+0000 7f9ab42c3700 1 -- 172.21.15.59:0/1883048017 wait complete. 2023-12-26T21:56:00.252 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.249+0000 7f9ab42c3700 1 Processor -- start 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.249+0000 7f9ab42c3700 1 -- start start 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab42c3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac10d9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab42c3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9aac0faf00 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab205f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac10d9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab205f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac10d9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:42822/0 (socket says 172.21.15.59:42822) 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab205f700 1 -- 172.21.15.59:0/3725776849 learned_addr learned my addr 172.21.15.59:0/3725776849 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab205f700 1 -- 172.21.15.59:0/3725776849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9aa800b200 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab205f700 1 --2- 172.21.15.59:0/3725776849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac10d9b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9aa8006800 tx=0x7f9aa8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa800de50 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f9aa801b870 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9aac0fb100 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9aa801a7a0 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.250+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9aac10e0e0 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.253 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.251+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f9aa800f070 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.251+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f9aa80247a0 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.252+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9aac045bb0 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.254 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.254+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9aa801f030 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.320+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f9aac06b250 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.321 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.321+0000 7f9a9effd700 1 -- 172.21.15.59:0/3725776849 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+175 (secure 0 0 0) 0x7f9aa8033090 con 0x7f9aac0fb4e0 2023-12-26T21:56:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.321+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 msgr2=0x7f9aac10d9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.321+0000 7f9ab42c3700 1 --2- 172.21.15.59:0/3725776849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac10d9b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f9aa8006800 tx=0x7f9aa8006ee0).stop 2023-12-26T21:56:00.322 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.321+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 shutdown_connections 2023-12-26T21:56:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.321+0000 7f9ab42c3700 1 --2- 172.21.15.59:0/3725776849 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9aac0fb4e0 0x7f9aac10d9b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.321+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 >> 172.21.15.59:0/3725776849 conn(0x7f9aac0f5df0 msgr2=0x7f9aac0fe430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.322+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 shutdown_connections 2023-12-26T21:56:00.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:00.322+0000 7f9ab42c3700 1 -- 172.21.15.59:0/3725776849 wait complete. 2023-12-26T21:56:00.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: audit 2023-12-26T21:55:59.523664+0000 mon.smithi059 (mon.0) 8 : audit [INF] from='client.? 172.21.15.59:0/111839057' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-26T21:56:00.506 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: audit 2023-12-26T21:55:59.526716+0000 mon.smithi059 (mon.0) 9 : audit [INF] from='client.? 172.21.15.59:0/111839057' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-26T21:56:00.666 INFO:teuthology.orchestra.run.smithi059.stderr:Restarting the monitor... 2023-12-26T21:56:00.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:00 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for 761f1ff0-a439-11ee-95a5-87774f69a715... 2023-12-26T21:56:00.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:00 smithi059 bash[111758]: Error: no container with name or ID "ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon.smithi059" found: no such container 2023-12-26T21:56:01.147 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: debug 2023-12-26T21:56:00.863+0000 7efdc8394700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-26T21:56:01.147 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111437]: debug 2023-12-26T21:56:00.863+0000 7efdc8394700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2023-12-26T21:56:01.147 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 bash[111797]: ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059 2023-12-26T21:56:01.544 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 systemd[1]: ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059.service: Succeeded. 2023-12-26T21:56:01.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 systemd[1]: Stopped Ceph mon.smithi059 for 761f1ff0-a439-11ee-95a5-87774f69a715. 2023-12-26T21:56:01.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 systemd[1]: Starting Ceph mon.smithi059 for 761f1ff0-a439-11ee-95a5-87774f69a715... 2023-12-26T21:56:01.725 INFO:teuthology.orchestra.run.smithi059.stderr:Setting public_network to 172.21.0.0/20 in mon config section 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.712+0000 7ff63946d700 0 set uid:gid to 167:167 (ceph:ceph) 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.712+0000 7ff63946d700 0 ceph version 16.2.5 (0883bdea7337b95e4b611c768c0279868462204a) pacific (stable), process ceph-mon, pid 7 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.712+0000 7ff63946d700 0 pidfile_write: ignore empty --pid-file 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.716+0000 7ff63946d700 0 load: jerasure load: lrc load: isa 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: RocksDB version: 6.8.1 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Compile date Jul 8 2021 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: DB SUMMARY 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: CURRENT file: CURRENT 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: IDENTITY file: IDENTITY 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi059/store.db dir, Total Num: 1, files: 000004.sst 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi059/store.db: 000006.log size: 94694 ; 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.error_if_exists: 0 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.create_if_missing: 0 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.paranoid_checks: 1 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.env: 0x5604667d21c0 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.fs: Posix File System 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.info_log: 0x5604671f3a40 2023-12-26T21:56:02.047 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_file_opening_threads: 16 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.statistics: (nil) 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.use_fsync: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_log_file_size: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.keep_log_file_num: 1000 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.recycle_log_file_num: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.allow_fallocate: 1 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.allow_mmap_reads: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.allow_mmap_writes: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.use_direct_reads: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.create_missing_column_families: 0 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.db_log_dir: 2023-12-26T21:56:02.048 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi059/store.db 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_subcompactions: 1 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_background_flushes: -1 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.advise_random_on_open: 1 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.db_write_buffer_size: 0 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.write_buffer_manager: 0x560467e6a030 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-12-26T21:56:02.049 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.rate_limiter: (nil) 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.wal_recovery_mode: 2 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.enable_thread_tracking: 0 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.enable_pipelined_write: 0 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.unordered_write: 0 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.row_cache: None 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.wal_filter: None 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-12-26T21:56:02.050 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.allow_ingest_behind: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.preserve_deletes: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.two_write_queues: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.manual_wal_flush: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.atomic_flush: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.log_readahead_size: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_background_jobs: 2 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_background_compactions: -1 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 bash[111916]: 7ad791493ac587090f9bc987ec30f55874028fde60ab0e1aa027be19d2e33aac 2023-12-26T21:56:02.051 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_total_wal_size: 0 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_open_files: -1 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bytes_per_sync: 0 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_readahead_size: 0 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Compression algorithms supported: 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-12-26T21:56:02.052 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kZSTD supported: 0 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kXpressCompression supported: 0 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kLZ4HCCompression supported: 1 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kLZ4Compression supported: 1 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kBZip2Compression supported: 0 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kZlibCompression supported: 1 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: kSnappyCompression supported: 1 2023-12-26T21:56:02.053 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-12-26T21:56:02.054 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000005 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.merge_operator: 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_filter: None 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_filter_factory: None 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x56046710dcc0) 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cache_index_and_filter_blocks: 1 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cache_index_and_filter_blocks_with_high_priority: 0 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-12-26T21:56:02.055 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: pin_top_level_index_and_filter: 1 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: index_type: 0 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: data_block_index_type: 0 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: index_shortening: 1 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: data_block_hash_table_util_ratio: 0.750000 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: hash_index_allow_collision: 1 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: checksum: 1 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: no_block_cache: 0 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_cache: 0x560467142f10 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_cache_name: BinnedLRUCache 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_cache_options: 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: capacity : 536870912 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: num_shard_bits : 4 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: strict_capacity_limit : 0 2023-12-26T21:56:02.056 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: high_pri_pool_ratio: 0.000 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_cache_compressed: (nil) 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: persistent_cache: (nil) 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_size: 4096 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_size_deviation: 10 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_restart_interval: 16 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: index_block_restart_interval: 1 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: metadata_block_size: 4096 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: partition_filters: 0 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: use_delta_encoding: 1 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: filter_policy: rocksdb.BuiltinBloomFilter 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: whole_key_filtering: 1 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: verify_compression: 0 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: read_amp_bytes_per_bit: 0 2023-12-26T21:56:02.057 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: format_version: 2 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: enable_index_compression: 1 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: block_align: 0 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.write_buffer_size: 33554432 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_write_buffer_number: 2 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression: NoCompression 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression: Disabled 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.prefix_extractor: nullptr 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.num_levels: 7 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-12-26T21:56:02.058 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression_opts.level: 32767 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression_opts.strategy: 0 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compression_opts.enabled: false 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-12-26T21:56:02.059 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.target_file_size_base: 67108864 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-12-26T21:56:02.060 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.arena_block_size: 4194304 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.disable_auto_compactions: 0 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-12-26T21:56:02.061 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.table_properties_collectors: 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.inplace_update_support: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.bloom_locality: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.max_successive_merges: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.paranoid_file_checks: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.force_consistency_checks: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.report_bg_io_stats: 0 2023-12-26T21:56:02.062 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.ttl: 2592000 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.717+0000 7ff63946d700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.718+0000 7ff63946d700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi059/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.718+0000 7ff63946d700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.718+0000 7ff63946d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703627761719404, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.718+0000 7ff63946d700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.718+0000 7ff63946d700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.719+0000 7ff63946d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703627761720326, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 90210, "table_properties": {"data_size": 88332, "index_size": 355, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12870, "raw_average_key_size": 50, "raw_value_size": 81284, "raw_average_value_size": 316, "num_data_blocks": 11, "num_entries": 257, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1703627761, "oldest_key_time": 3, "file_creation_time": 0}} 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.719+0000 7ff63946d700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.719+0000 7ff63946d700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1703627761720722, "job": 1, "event": "recovery_finished"} 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.720+0000 7ff63946d700 4 rocksdb: DB pointer 0x560467207800 2023-12-26T21:56:02.063 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.720+0000 7ff620275700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.720+0000 7ff620275700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** DB Stats ** 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** Compaction Stats [default] ** 2023-12-26T21:56:02.064 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: L0 2/0 89.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Sum 2/0 89.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** Compaction Stats [default] ** 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-26T21:56:02.065 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(Total Files): cumulative 0, interval 0 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(Keys): cumulative 0, interval 0 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Cumulative compaction: 0.00 GB write, 30.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Interval compaction: 0.00 GB write, 30.99 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** File Read Latency Histogram By Level [default] ** 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** Compaction Stats [default] ** 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: L0 2/0 89.79 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Sum 2/0 89.79 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.066 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** Compaction Stats [default] ** 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 143.9 0.00 0.00 1 0.001 0 0 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Uptime(secs): 0.0 total, 0.0 interval 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Flush(GB): cumulative 0.000, interval 0.000 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(GB): cumulative 0.000, interval 0.000 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(Total Files): cumulative 0, interval 0 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(L0 Files): cumulative 0, interval 0 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: AddFile(Keys): cumulative 0, interval 0 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Cumulative compaction: 0.00 GB write, 30.67 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-12-26T21:56:02.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ** File Read Latency Histogram By Level [default] ** 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.721+0000 7ff63946d700 0 starting mon.smithi059 rank 0 at public addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] at bind addrs [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi059 fsid 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 1 mon.smithi059@-1(???) e1 preinit fsid 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 0 mon.smithi059@-1(???).mds e1 new map 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 0 mon.smithi059@-1(???).mds e1 print_map 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: e1 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: enable_multiple, ever_enabled_multiple: 1,1 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: legacy client fscid: -1 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: No filesystems configured 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 systemd[1]: Started Ceph mon.smithi059 for 761f1ff0-a439-11ee-95a5-87774f69a715. 2023-12-26T21:56:02.068 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 0 mon.smithi059@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 0 mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 0 mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.722+0000 7ff63946d700 0 mon.smithi059@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:55:59.523664+0000 mon.smithi059 (mon.0) 8 : audit [INF] from='client.? 172.21.15.59:0/111839057' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:55:59.526716+0000 mon.smithi059 (mon.0) 9 : audit [INF] from='client.? 172.21.15.59:0/111839057' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.723+0000 7ff63946d700 1 mon.smithi059@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expand map: {default=false} 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta from 'false' to 'false' 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expanded map: {default=false} 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expand map: {default=info} 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta from 'info' to 'info' 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expanded map: {default=info} 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expand map: {default=daemon} 2023-12-26T21:56:02.069 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expanded map: {default=daemon} 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expand map: {default=debug} 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta from 'debug' to 'debug' 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T21:56:01.724+0000 7ff63946d700 20 expand_channel_meta expanded map: {default=debug} 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:01.739916+0000 mon.smithi059 (mon.0) 1 : cluster [INF] mon.smithi059 is new leader, mons smithi059 in quorum (ranks 0) 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:01.739981+0000 mon.smithi059 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi059=[v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0]} 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:01.740025+0000 mon.smithi059 (mon.0) 3 : cluster [DBG] fsmap 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:01.740053+0000 mon.smithi059 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-12-26T21:56:02.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:01.740482+0000 mon.smithi059 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.066+0000 7fc29c23b700 1 Processor -- start 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.067+0000 7fc29c23b700 1 -- start start 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.067+0000 7fc29c23b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc2940fec00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.067+0000 7fc29c23b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2940ff0f0 con 0x7fc29406adc0 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.067+0000 7fc299fd7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc2940fec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.067+0000 7fc299fd7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc2940fec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34446/0 (socket says 172.21.15.59:34446) 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.067+0000 7fc299fd7700 1 -- 172.21.15.59:0/563342851 learned_addr learned my addr 172.21.15.59:0/563342851 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:02.073 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc299fd7700 1 -- 172.21.15.59:0/563342851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2940ff230 con 0x7fc29406adc0 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc299fd7700 1 --2- 172.21.15.59:0/563342851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc2940fec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc28c009960 tx=0x7fc28c0092d0).ready entity=mon.0 client_cookie=35f2778c2b442ec0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc298fd5700 1 -- 172.21.15.59:0/563342851 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc28c00b9c0 con 0x7fc29406adc0 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc298fd5700 1 -- 172.21.15.59:0/563342851 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc28c011400 con 0x7fc29406adc0 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc298fd5700 1 -- 172.21.15.59:0/563342851 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc28c010580 con 0x7fc29406adc0 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc29c23b700 1 -- 172.21.15.59:0/563342851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 msgr2=0x7fc2940fec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.068+0000 7fc29c23b700 1 --2- 172.21.15.59:0/563342851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc2940fec00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc28c009960 tx=0x7fc28c0092d0).stop 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.069+0000 7fc29c23b700 1 -- 172.21.15.59:0/563342851 shutdown_connections 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.069+0000 7fc29c23b700 1 --2- 172.21.15.59:0/563342851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc2940fec00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.069+0000 7fc29c23b700 1 -- 172.21.15.59:0/563342851 >> 172.21.15.59:0/563342851 conn(0x7fc2940f6680 msgr2=0x7fc2940f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.069+0000 7fc29c23b700 1 -- 172.21.15.59:0/563342851 shutdown_connections 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.069+0000 7fc29c23b700 1 -- 172.21.15.59:0/563342851 wait complete. 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.069+0000 7fc29c23b700 1 Processor -- start 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc29c23b700 1 -- start start 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc29c23b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc29410ad90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:02.074 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc29c23b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29410b280 con 0x7fc29406adc0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc299fd7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc29410ad90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc299fd7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc29410ad90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34456/0 (socket says 172.21.15.59:34456) 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc299fd7700 1 -- 172.21.15.59:0/3863468901 learned_addr learned my addr 172.21.15.59:0/3863468901 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc299fd7700 1 -- 172.21.15.59:0/3863468901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc28c009570 con 0x7fc29406adc0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.070+0000 7fc299fd7700 1 --2- 172.21.15.59:0/3863468901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc29410ad90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc28c011c40 tx=0x7fc28c0092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc28c010840 con 0x7fc29406adc0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc28c010e00 con 0x7fc29406adc0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29410b480 con 0x7fc29406adc0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc28c01acf0 con 0x7fc29406adc0 2023-12-26T21:56:02.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29410c0c0 con 0x7fc29406adc0 2023-12-26T21:56:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7fc28c02a070 con 0x7fc29406adc0 2023-12-26T21:56:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.071+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc28c0223b0 con 0x7fc29406adc0 2023-12-26T21:56:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.072+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc29410b910 con 0x7fc29406adc0 2023-12-26T21:56:02.076 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.074+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fc28c028060 con 0x7fc29406adc0 2023-12-26T21:56:02.140 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.140+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fc2940591e0 con 0x7fc29406adc0 2023-12-26T21:56:02.145 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc2927fc700 1 -- 172.21.15.59:0/3863468901 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fc28c01e070 con 0x7fc29406adc0 2023-12-26T21:56:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 msgr2=0x7fc29410ad90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 --2- 172.21.15.59:0/3863468901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc29410ad90 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc28c011c40 tx=0x7fc28c0092d0).stop 2023-12-26T21:56:02.146 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 shutdown_connections 2023-12-26T21:56:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 --2- 172.21.15.59:0/3863468901 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc29406adc0 0x7fc29410ad90 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 >> 172.21.15.59:0/3863468901 conn(0x7fc2940f6680 msgr2=0x7fc2940f8190 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 shutdown_connections 2023-12-26T21:56:02.147 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:02.145+0000 7fc29c23b700 1 -- 172.21.15.59:0/3863468901 wait complete. 2023-12-26T21:56:02.490 INFO:teuthology.orchestra.run.smithi059.stderr:Wrote config to /etc/ceph/ceph.conf 2023-12-26T21:56:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-12-26T21:56:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:Creating mgr... 2023-12-26T21:56:02.491 INFO:teuthology.orchestra.run.smithi059.stderr:Verifying port 9283 ... 2023-12-26T21:56:02.626 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Failed to reset failed state of unit ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mgr.smithi059.gwkiyc.service: Unit ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mgr.smithi059.gwkiyc.service not loaded. 2023-12-26T21:56:02.632 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: Created symlink /etc/systemd/system/ceph-761f1ff0-a439-11ee-95a5-87774f69a715.target.wants/ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mgr.smithi059.gwkiyc.service → /etc/systemd/system/ceph-761f1ff0-a439-11ee-95a5-87774f69a715@.service. 2023-12-26T21:56:03.192 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2023-12-26T21:56:03.200 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2023-12-26T21:56:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2023-12-26T21:56:03.201 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to enable service . firewalld.service is not available 2023-12-26T21:56:03.209 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2023-12-26T21:56:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2023-12-26T21:56:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2023-12-26T21:56:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-12-26T21:56:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr to start... 2023-12-26T21:56:03.219 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr... 2023-12-26T21:56:03.544 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:03 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:02.146072+0000 mon.smithi059 (mon.0) 6 : audit [INF] from='client.? 172.21.15.59:0/3863468901' entity='client.admin' 2023-12-26T21:56:03.578 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.577+0000 7ff3e759e700 1 Processor -- start 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.577+0000 7ff3e759e700 1 -- start start 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.577+0000 7ff3e759e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e80fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.577+0000 7ff3e759e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e80fef20 con 0x7ff3e806ade0 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e80fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e80fea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34486/0 (socket says 172.21.15.59:34486) 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e659c700 1 -- 172.21.15.59:0/1325665792 learned_addr learned my addr 172.21.15.59:0/1325665792 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e659c700 1 -- 172.21.15.59:0/1325665792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e80ff060 con 0x7ff3e806ade0 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e659c700 1 --2- 172.21.15.59:0/1325665792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e80fea30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff3d800b5f0 tx=0x7ff3d8006ee0).ready entity=mon.0 client_cookie=f80acc27b99b4c21 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e559a700 1 -- 172.21.15.59:0/1325665792 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d800d600 con 0x7ff3e806ade0 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e559a700 1 -- 172.21.15.59:0/1325665792 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff3d800dbc0 con 0x7ff3e806ade0 2023-12-26T21:56:03.580 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.578+0000 7ff3e559a700 1 -- 172.21.15.59:0/1325665792 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d8011b90 con 0x7ff3e806ade0 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 -- 172.21.15.59:0/1325665792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 msgr2=0x7ff3e80fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 --2- 172.21.15.59:0/1325665792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e80fea30 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7ff3d800b5f0 tx=0x7ff3d8006ee0).stop 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 -- 172.21.15.59:0/1325665792 shutdown_connections 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 --2- 172.21.15.59:0/1325665792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e80fea30 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 -- 172.21.15.59:0/1325665792 >> 172.21.15.59:0/1325665792 conn(0x7ff3e80f64d0 msgr2=0x7ff3e80f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 -- 172.21.15.59:0/1325665792 shutdown_connections 2023-12-26T21:56:03.581 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.579+0000 7ff3e759e700 1 -- 172.21.15.59:0/1325665792 wait complete. 2023-12-26T21:56:03.582 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e759e700 1 Processor -- start 2023-12-26T21:56:03.582 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e759e700 1 -- start start 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e759e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e810ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e759e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3e810b010 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e810ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e810ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34494/0 (socket says 172.21.15.59:34494) 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e659c700 1 -- 172.21.15.59:0/3431371033 learned_addr learned my addr 172.21.15.59:0/3431371033 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.580+0000 7ff3e659c700 1 -- 172.21.15.59:0/3431371033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3d800b200 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3e659c700 1 --2- 172.21.15.59:0/3431371033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e810ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff3d8006800 tx=0x7ff3d8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d800de50 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7ff3d801b870 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3e810b210 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3d801a7a0 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3e810b6a0 con 0x7ff3e806ade0 2023-12-26T21:56:03.583 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7ff3d800f070 con 0x7ff3e806ade0 2023-12-26T21:56:03.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.581+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7ff3d8024bc0 con 0x7ff3e806ade0 2023-12-26T21:56:03.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.582+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3e8045bb0 con 0x7ff3e806ade0 2023-12-26T21:56:03.584 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.584+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7ff3d801f030 con 0x7ff3e806ade0 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.654+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7ff3e80ffd60 con 0x7ff3e806ade0 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.654+0000 7ff3d6ffd700 1 -- 172.21.15.59:0/3431371033 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7ff3d8033090 con 0x7ff3e806ade0 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsid": "761f1ff0-a439-11ee-95a5-87774f69a715", 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "health": { 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-26T21:56:03.660 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "checks": {}, 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mutes": [] 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum": [ 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 0 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "smithi059" 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "monmap": { 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-26T21:56:03.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osdmap": { 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgmap": { 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-26T21:56:03.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsmap": { 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": false, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modules": [ 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "iostat", 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "nfs", 2023-12-26T21:56:03.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "restful" 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "servicemap": { 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modified": "2023-12-26T21:55:58.161436+0000", 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 msgr2=0x7ff3e810ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 --2- 172.21.15.59:0/3431371033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e810ab20 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7ff3d8006800 tx=0x7ff3d8006ee0).stop 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 shutdown_connections 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 --2- 172.21.15.59:0/3431371033 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff3e806ade0 0x7ff3e810ab20 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:03.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 >> 172.21.15.59:0/3431371033 conn(0x7ff3e80f64d0 msgr2=0x7ff3e80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:03.665 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 shutdown_connections 2023-12-26T21:56:03.665 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:03.655+0000 7ff3e759e700 1 -- 172.21.15.59:0/3431371033 wait complete. 2023-12-26T21:56:03.963 INFO:teuthology.orchestra.run.smithi059.stderr:mgr not available, waiting (1/15)... 2023-12-26T21:56:04.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:03.655421+0000 mon.smithi059 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.59:0/3431371033' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e970c700 1 Processor -- start 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e970c700 1 -- start start 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e970c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40feab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e970c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40fefa0 con 0x7f29e406ad60 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40feab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40feab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34506/0 (socket says 172.21.15.59:34506) 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.362+0000 7f29e3fff700 1 -- 172.21.15.59:0/63834350 learned_addr learned my addr 172.21.15.59:0/63834350 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e3fff700 1 -- 172.21.15.59:0/63834350 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e40ff0e0 con 0x7f29e406ad60 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e3fff700 1 --2- 172.21.15.59:0/63834350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40feab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f29d400b5f0 tx=0x7f29d4006ee0).ready entity=mon.0 client_cookie=e5255116d54bb12f server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:06.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e2ffd700 1 -- 172.21.15.59:0/63834350 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d400d600 con 0x7f29e406ad60 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e2ffd700 1 -- 172.21.15.59:0/63834350 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f29d400dbc0 con 0x7f29e406ad60 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e2ffd700 1 -- 172.21.15.59:0/63834350 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d4011b90 con 0x7f29e406ad60 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e970c700 1 -- 172.21.15.59:0/63834350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 msgr2=0x7f29e40feab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e970c700 1 --2- 172.21.15.59:0/63834350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40feab0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f29d400b5f0 tx=0x7f29d4006ee0).stop 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e970c700 1 -- 172.21.15.59:0/63834350 shutdown_connections 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e970c700 1 --2- 172.21.15.59:0/63834350 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40feab0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.363+0000 7f29e970c700 1 -- 172.21.15.59:0/63834350 >> 172.21.15.59:0/63834350 conn(0x7f29e40f64d0 msgr2=0x7f29e40f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.364+0000 7f29e970c700 1 -- 172.21.15.59:0/63834350 shutdown_connections 2023-12-26T21:56:06.366 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.364+0000 7f29e970c700 1 -- 172.21.15.59:0/63834350 wait complete. 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.364+0000 7f29e970c700 1 Processor -- start 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.364+0000 7f29e970c700 1 -- start start 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.364+0000 7f29e970c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40fc530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.364+0000 7f29e970c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29e40fa9b0 con 0x7f29e406ad60 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40fc530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40fc530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34518/0 (socket says 172.21.15.59:34518) 2023-12-26T21:56:06.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e3fff700 1 -- 172.21.15.59:0/2239411620 learned_addr learned my addr 172.21.15.59:0/2239411620 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e3fff700 1 -- 172.21.15.59:0/2239411620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29d400b200 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e3fff700 1 --2- 172.21.15.59:0/2239411620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40fc530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29d4000c00 tx=0x7f29d4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d400de00 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f29d401b870 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29e40fab50 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29d401a670 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.365+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29e40fafe0 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.366+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 1) v1 ==== 491+0+0 (secure 0 0 0) 0x7f29d402b070 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.366+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f29d4024a50 con 0x7f29e406ad60 2023-12-26T21:56:06.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.366+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29e4045bb0 con 0x7f29e406ad60 2023-12-26T21:56:06.369 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.368+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f29d401f020 con 0x7f29e406ad60 2023-12-26T21:56:06.439 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.438+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f29e40008d0 con 0x7f29e406ad60 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.438+0000 7f29e0ff9700 1 -- 172.21.15.59:0/2239411620 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1249 (secure 0 0 0) 0x7f29d4016070 con 0x7f29e406ad60 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsid": "761f1ff0-a439-11ee-95a5-87774f69a715", 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "health": { 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "checks": {}, 2023-12-26T21:56:06.440 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mutes": [] 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum": [ 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 0 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "smithi059" 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "monmap": { 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-26T21:56:06.443 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osdmap": { 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgmap": { 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-26T21:56:06.444 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-26T21:56:06.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-26T21:56:06.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-26T21:56:06.445 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsmap": { 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": false, 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modules": [ 2023-12-26T21:56:06.448 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "iostat", 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "nfs", 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "restful" 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "servicemap": { 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modified": "2023-12-26T21:55:58.161436+0000", 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 msgr2=0x7f29e40fc530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 --2- 172.21.15.59:0/2239411620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40fc530 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f29d4000c00 tx=0x7f29d4006ee0).stop 2023-12-26T21:56:06.449 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 shutdown_connections 2023-12-26T21:56:06.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 --2- 172.21.15.59:0/2239411620 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29e406ad60 0x7f29e40fc530 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:06.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 >> 172.21.15.59:0/2239411620 conn(0x7f29e40f64d0 msgr2=0x7f29e40f7fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:06.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 shutdown_connections 2023-12-26T21:56:06.450 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:06.439+0000 7f29e970c700 1 -- 172.21.15.59:0/2239411620 wait complete. 2023-12-26T21:56:06.725 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:06.439832+0000 mon.smithi059 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.59:0/2239411620' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-26T21:56:06.790 INFO:teuthology.orchestra.run.smithi059.stderr:mgr not available, waiting (2/15)... 2023-12-26T21:56:09.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:08.543251+0000 mon.smithi059 (mon.0) 9 : cluster [INF] Activating manager daemon smithi059.gwkiyc 2023-12-26T21:56:09.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:08.545684+0000 mon.smithi059 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi059.gwkiyc(active, starting, since 0.00266504s) 2023-12-26T21:56:09.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.546647+0000 mon.smithi059 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.546851+0000 mon.smithi059 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.547040+0000 mon.smithi059 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.547271+0000 mon.smithi059 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.547484+0000 mon.smithi059 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.gwkiyc", "id": "smithi059.gwkiyc"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:08.560475+0000 mon.smithi059 (mon.0) 16 : cluster [INF] Manager daemon smithi059.gwkiyc is now available 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.568974+0000 mon.smithi059 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.573888+0000 mon.smithi059 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.575091+0000 mon.smithi059 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.579896+0000 mon.smithi059 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:09.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:08.582177+0000 mon.smithi059 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.59:0/436875376' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.460+0000 7fc69f04c700 1 Processor -- start 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.460+0000 7fc69f04c700 1 -- start start 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.461+0000 7fc69f04c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.461+0000 7fc69f04c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6980ff0d0 con 0x7fc698027670 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.461+0000 7fc69cde8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.461+0000 7fc69cde8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34574/0 (socket says 172.21.15.59:34574) 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.461+0000 7fc69cde8700 1 -- 172.21.15.59:0/3773944902 learned_addr learned my addr 172.21.15.59:0/3773944902 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.461+0000 7fc69cde8700 1 -- 172.21.15.59:0/3773944902 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6980ff210 con 0x7fc698027670 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69cde8700 1 --2- 172.21.15.59:0/3773944902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980febe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc68c00b5f0 tx=0x7fc68c006ee0).ready entity=mon.0 client_cookie=c97761c3538c6a82 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc6977fe700 1 -- 172.21.15.59:0/3773944902 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c00d600 con 0x7fc698027670 2023-12-26T21:56:10.557 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc6977fe700 1 -- 172.21.15.59:0/3773944902 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc68c00dbc0 con 0x7fc698027670 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc6977fe700 1 -- 172.21.15.59:0/3773944902 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c011b90 con 0x7fc698027670 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69f04c700 1 -- 172.21.15.59:0/3773944902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 msgr2=0x7fc6980febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69f04c700 1 --2- 172.21.15.59:0/3773944902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980febe0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fc68c00b5f0 tx=0x7fc68c006ee0).stop 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69f04c700 1 -- 172.21.15.59:0/3773944902 shutdown_connections 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69f04c700 1 --2- 172.21.15.59:0/3773944902 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980febe0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69f04c700 1 -- 172.21.15.59:0/3773944902 >> 172.21.15.59:0/3773944902 conn(0x7fc6980f6680 msgr2=0x7fc6980f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.462+0000 7fc69f04c700 1 -- 172.21.15.59:0/3773944902 shutdown_connections 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.463+0000 7fc69f04c700 1 -- 172.21.15.59:0/3773944902 wait complete. 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.463+0000 7fc69f04c700 1 Processor -- start 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.463+0000 7fc69f04c700 1 -- start start 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.463+0000 7fc69f04c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980fc660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.463+0000 7fc69f04c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6980faae0 con 0x7fc698027670 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69cde8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980fc660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69cde8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980fc660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:34584/0 (socket says 172.21.15.59:34584) 2023-12-26T21:56:10.558 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69cde8700 1 -- 172.21.15.59:0/2862114329 learned_addr learned my addr 172.21.15.59:0/2862114329 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69cde8700 1 -- 172.21.15.59:0/2862114329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc68c00b200 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69cde8700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980fc660 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc68c006800 tx=0x7fc68c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c00de50 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fc68c01b870 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6980fac80 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc68c01a7a0 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.464+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6980fb110 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.465+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 3) v1 ==== 37004+0+0 (secure 0 0 0) 0x7fc68c00f070 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.465+0000 7fc6957fa700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fc67c030f00 0x7fc67c0333c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.465+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fc68c0429b0 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.465+0000 7fc697fff700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fc67c030f00 0x7fc67c0333c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.465+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6980591e0 con 0x7fc698027670 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.466+0000 7fc697fff700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fc67c030f00 0x7fc67c0333c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc6840096a0 tx=0x7fc684006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:10.559 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.470+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc68c016030 con 0x7fc698027670 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.695+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fc698045bb0 con 0x7fc698027670 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.695+0000 7fc6957fa700 1 -- 172.21.15.59:0/2862114329 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fc68c032090 con 0x7fc698027670 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsid": "761f1ff0-a439-11ee-95a5-87774f69a715", 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "health": { 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "checks": {}, 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mutes": [] 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.704 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum": [ 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 0 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_names": [ 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "smithi059" 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "quorum_age": 8, 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "monmap": { 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_mons": 1 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osdmap": { 2023-12-26T21:56:10.705 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_osds": 0, 2023-12-26T21:56:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-12-26T21:56:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-12-26T21:56:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-12-26T21:56:10.706 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgmap": { 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_pools": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_objects": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "bytes_total": 0 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.707 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "fsmap": { 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "by_rank": [], 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "up:standby": 0 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap": { 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": true, 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modules": [ 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "iostat", 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "nfs", 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "restful" 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ], 2023-12-26T21:56:10.708 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "servicemap": { 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 1, 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "modified": "2023-12-26T21:55:58.161436+0000", 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "services": {} 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: }, 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "progress_events": {} 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fc67c030f00 msgr2=0x7fc67c0333c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fc67c030f00 0x7fc67c0333c0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7fc6840096a0 tx=0x7fc684006b40).stop 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 msgr2=0x7fc6980fc660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980fc660 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7fc68c006800 tx=0x7fc68c006ee0).stop 2023-12-26T21:56:10.709 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 shutdown_connections 2023-12-26T21:56:10.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fc67c030f00 0x7fc67c0333c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:10.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 --2- 172.21.15.59:0/2862114329 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc698027670 0x7fc6980fc660 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:10.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 >> 172.21.15.59:0/2862114329 conn(0x7fc6980f6680 msgr2=0x7fc6980f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:10.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 shutdown_connections 2023-12-26T21:56:10.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:10.697+0000 7fc69f04c700 1 -- 172.21.15.59:0/2862114329 wait complete. 2023-12-26T21:56:11.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:09.549579+0000 mon.smithi059 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi059.gwkiyc(active, since 1.00655s) 2023-12-26T21:56:11.725 INFO:teuthology.orchestra.run.smithi059.stderr:mgr is available 2023-12-26T21:56:12.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:11 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:10.696651+0000 mon.smithi059 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.59:0/2862114329' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-12-26T21:56:12.625 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:11.549437+0000 mon.smithi059 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi059.gwkiyc(active, since 3s) 2023-12-26T21:56:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb50ab8700 1 Processor -- start 2023-12-26T21:56:13.376 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb50ab8700 1 -- start start 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb50ab8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c100fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb50ab8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c0fc9d0 con 0x7fcb4c027670 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb4a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c100fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb4a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c100fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38652/0 (socket says 172.21.15.59:38652) 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.194+0000 7fcb4a59c700 1 -- 172.21.15.59:0/3837550792 learned_addr learned my addr 172.21.15.59:0/3837550792 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb4a59c700 1 -- 172.21.15.59:0/3837550792 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb4c1014d0 con 0x7fcb4c027670 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb4a59c700 1 --2- 172.21.15.59:0/3837550792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c100fe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcb3c00b5f0 tx=0x7fcb3c006ee0).ready entity=mon.0 client_cookie=e3668b38b4e5e339 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb4959a700 1 -- 172.21.15.59:0/3837550792 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c00d600 con 0x7fcb4c027670 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb4959a700 1 -- 172.21.15.59:0/3837550792 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcb3c00dbc0 con 0x7fcb4c027670 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb4959a700 1 -- 172.21.15.59:0/3837550792 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c011b90 con 0x7fcb4c027670 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 -- 172.21.15.59:0/3837550792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 msgr2=0x7fcb4c100fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:13.377 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 --2- 172.21.15.59:0/3837550792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c100fe0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fcb3c00b5f0 tx=0x7fcb3c006ee0).stop 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 -- 172.21.15.59:0/3837550792 shutdown_connections 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 --2- 172.21.15.59:0/3837550792 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c100fe0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 -- 172.21.15.59:0/3837550792 >> 172.21.15.59:0/3837550792 conn(0x7fcb4c0f6680 msgr2=0x7fcb4c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 -- 172.21.15.59:0/3837550792 shutdown_connections 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.195+0000 7fcb50ab8700 1 -- 172.21.15.59:0/3837550792 wait complete. 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb50ab8700 1 Processor -- start 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb50ab8700 1 -- start start 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb50ab8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c10cec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb50ab8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c10d3b0 con 0x7fcb4c027670 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb4a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c10cec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb4a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c10cec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38662/0 (socket says 172.21.15.59:38662) 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.196+0000 7fcb4a59c700 1 -- 172.21.15.59:0/1158094529 learned_addr learned my addr 172.21.15.59:0/1158094529 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:13.378 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb4a59c700 1 -- 172.21.15.59:0/1158094529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c00b200 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb4a59c700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c10cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fcb3c000c00 tx=0x7fcb3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c01b410 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fcb3c01b9d0 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c10d5b0 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c01a620 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.197+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c10da40 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.198+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7fcb3c02b070 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.198+0000 7fcb42ffd700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fcb2c030fa0 0x7fcb2c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.198+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcb3c042ac0 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.198+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c045bb0 con 0x7fcb4c027670 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.198+0000 7fcb49d9b700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fcb2c030fa0 0x7fcb2c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.198+0000 7fcb49d9b700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fcb2c030fa0 0x7fcb2c033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcb340096a0 tx=0x7fcb34006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:13.379 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.202+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcb3c01f080 con 0x7fcb4c027670 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.364+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7fcb4c102110 con 0x7fcb4c027670 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.365+0000 7fcb42ffd700 1 -- 172.21.15.59:0/1158094529 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+398 (secure 0 0 0) 0x7fcb3c016020 con 0x7fcb4c027670 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fcb2c030fa0 msgr2=0x7fcb2c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fcb2c030fa0 0x7fcb2c033460 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fcb340096a0 tx=0x7fcb34006b40).stop 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 msgr2=0x7fcb4c10cec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c10cec0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fcb3c000c00 tx=0x7fcb3c006ee0).stop 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 shutdown_connections 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7fcb2c030fa0 0x7fcb2c033460 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 --2- 172.21.15.59:0/1158094529 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb4c027670 0x7fcb4c10cec0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.366+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 >> 172.21.15.59:0/1158094529 conn(0x7fcb4c0f6680 msgr2=0x7fcb4c0f8140 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.367+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 shutdown_connections 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.367+0000 7fcb50ab8700 1 -- 172.21.15.59:0/1158094529 wait complete. 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.380 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [global] 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: fsid = 761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: log_to_journald = false 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [mgr] 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: [osd] 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-12-26T21:56:13.381 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-12-26T21:56:13.382 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-12-26T21:56:13.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:13 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:13.365890+0000 mon.smithi059 (mon.0) 25 : audit [INF] from='client.? 172.21.15.59:0/1158094529' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-12-26T21:56:15.151 INFO:teuthology.orchestra.run.smithi059.stderr:Enabling cephadm module... 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.486+0000 7f19a49ce700 1 Processor -- start 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.486+0000 7f19a49ce700 1 -- start start 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.486+0000 7f19a49ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a00febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.486+0000 7f19a49ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a00ff0d0 con 0x7f19a006ade0 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.486+0000 7f199e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a00febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a00febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38666/0 (socket says 172.21.15.59:38666) 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199e59c700 1 -- 172.21.15.59:0/3344491004 learned_addr learned my addr 172.21.15.59:0/3344491004 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199e59c700 1 -- 172.21.15.59:0/3344491004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19a00ff210 con 0x7f19a006ade0 2023-12-26T21:56:16.506 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199e59c700 1 --2- 172.21.15.59:0/3344491004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a00febe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f198800b5f0 tx=0x7f1988006ee0).ready entity=mon.0 client_cookie=9305e90685f611bb server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199d59a700 1 -- 172.21.15.59:0/3344491004 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198800d600 con 0x7f19a006ade0 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199d59a700 1 -- 172.21.15.59:0/3344491004 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f198800dbc0 con 0x7f19a006ade0 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.487+0000 7f199d59a700 1 -- 172.21.15.59:0/3344491004 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1988011b90 con 0x7f19a006ade0 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 -- 172.21.15.59:0/3344491004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 msgr2=0x7f19a00febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 --2- 172.21.15.59:0/3344491004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a00febe0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f198800b5f0 tx=0x7f1988006ee0).stop 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 -- 172.21.15.59:0/3344491004 shutdown_connections 2023-12-26T21:56:16.507 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 --2- 172.21.15.59:0/3344491004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a00febe0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 -- 172.21.15.59:0/3344491004 >> 172.21.15.59:0/3344491004 conn(0x7f19a00f6680 msgr2=0x7f19a00f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 -- 172.21.15.59:0/3344491004 shutdown_connections 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 -- 172.21.15.59:0/3344491004 wait complete. 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 Processor -- start 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.488+0000 7f19a49ce700 1 -- start start 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f19a49ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a010ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f19a49ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19a010b230 con 0x7f19a006ade0 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f199e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a010ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f199e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a010ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38672/0 (socket says 172.21.15.59:38672) 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f199e59c700 1 -- 172.21.15.59:0/739059973 learned_addr learned my addr 172.21.15.59:0/739059973 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f199e59c700 1 -- 172.21.15.59:0/739059973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f198800b200 con 0x7f19a006ade0 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f199e59c700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a010ad40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f1988000c00 tx=0x7f1988006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198801b410 con 0x7f19a006ade0 2023-12-26T21:56:16.508 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f198801b9d0 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f198801a620 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19a010b430 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.489+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19a010c040 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.490+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 4) v1 ==== 37084+0+0 (secure 0 0 0) 0x7f198802b070 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.490+0000 7f1996ffd700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f1980030fa0 0x7f1980033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.490+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19a00591e0 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.490+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f1988042a80 con 0x7f19a006ade0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.490+0000 7f199dd9b700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f1980030fa0 0x7f1980033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.491+0000 7f199dd9b700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f1980030fa0 0x7f1980033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f19900096a0 tx=0x7f1990006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:16.509 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.495+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1988016020 con 0x7f19a006ade0 2023-12-26T21:56:16.698 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.698+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f19a010ce70 con 0x7f19a006ade0 2023-12-26T21:56:16.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.750+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f198800f070 con 0x7f19a006ade0 2023-12-26T21:56:16.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.751+0000 7f1996ffd700 1 -- 172.21.15.59:0/739059973 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f198802d9b0 con 0x7f19a006ade0 2023-12-26T21:56:16.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f1980030fa0 msgr2=0x7f1980033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:16.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f1980030fa0 0x7f1980033460 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f19900096a0 tx=0x7f1990006b40).stop 2023-12-26T21:56:16.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 msgr2=0x7f19a010ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a010ad40 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f1988000c00 tx=0x7f1988006ee0).stop 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 shutdown_connections 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f1980030fa0 0x7f1980033460 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 --2- 172.21.15.59:0/739059973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f19a006ade0 0x7f19a010ad40 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 >> 172.21.15.59:0/739059973 conn(0x7f19a00f6680 msgr2=0x7f19a00f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 shutdown_connections 2023-12-26T21:56:16.754 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:16.753+0000 7f19a49ce700 1 -- 172.21.15.59:0/739059973 wait complete. 2023-12-26T21:56:17.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:16.698604+0000 mon.smithi059 (mon.0) 26 : audit [INF] from='client.? 172.21.15.59:0/739059973' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-12-26T21:56:18.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:17 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:16.751418+0000 mon.smithi059 (mon.0) 27 : audit [INF] from='client.? 172.21.15.59:0/739059973' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-12-26T21:56:18.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:17 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:16.751469+0000 mon.smithi059 (mon.0) 28 : cluster [DBG] mgrmap e5: smithi059.gwkiyc(active, since 8s) 2023-12-26T21:56:18.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.518+0000 7f2990b64700 1 Processor -- start 2023-12-26T21:56:18.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.518+0000 7f2990b64700 1 -- start start 2023-12-26T21:56:18.527 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.519+0000 7f2990b64700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c0fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:18.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.519+0000 7f2990b64700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c0fcde0 con 0x7f298c0fc4d0 2023-12-26T21:56:18.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.519+0000 7f298a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c0fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:18.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.519+0000 7f298a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c0fc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38700/0 (socket says 172.21.15.59:38700) 2023-12-26T21:56:18.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.519+0000 7f298a59c700 1 -- 172.21.15.59:0/3355958656 learned_addr learned my addr 172.21.15.59:0/3355958656 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:18.528 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.519+0000 7f298a59c700 1 -- 172.21.15.59:0/3355958656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c0fcf20 con 0x7f298c0fc4d0 2023-12-26T21:56:18.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f298a59c700 1 --2- 172.21.15.59:0/3355958656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c0fc8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f297c00b5f0 tx=0x7f297c006ee0).ready entity=mon.0 client_cookie=54a6654c1df08965 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:18.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f298959a700 1 -- 172.21.15.59:0/3355958656 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c00d600 con 0x7f298c0fc4d0 2023-12-26T21:56:18.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f298959a700 1 -- 172.21.15.59:0/3355958656 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f297c00dbc0 con 0x7f298c0fc4d0 2023-12-26T21:56:18.529 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f298959a700 1 -- 172.21.15.59:0/3355958656 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c011b90 con 0x7f298c0fc4d0 2023-12-26T21:56:18.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f2990b64700 1 -- 172.21.15.59:0/3355958656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 msgr2=0x7f298c0fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:18.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f2990b64700 1 --2- 172.21.15.59:0/3355958656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c0fc8f0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f297c00b5f0 tx=0x7f297c006ee0).stop 2023-12-26T21:56:18.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f2990b64700 1 -- 172.21.15.59:0/3355958656 shutdown_connections 2023-12-26T21:56:18.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f2990b64700 1 --2- 172.21.15.59:0/3355958656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c0fc8f0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:18.530 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.520+0000 7f2990b64700 1 -- 172.21.15.59:0/3355958656 >> 172.21.15.59:0/3355958656 conn(0x7f298c0f5e20 msgr2=0x7f298c0f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:18.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f2990b64700 1 -- 172.21.15.59:0/3355958656 shutdown_connections 2023-12-26T21:56:18.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f2990b64700 1 -- 172.21.15.59:0/3355958656 wait complete. 2023-12-26T21:56:18.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f2990b64700 1 Processor -- start 2023-12-26T21:56:18.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f2990b64700 1 -- start start 2023-12-26T21:56:18.531 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f2990b64700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c10cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f2990b64700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f298c10d410 con 0x7f298c0fc4d0 2023-12-26T21:56:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f298a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c10cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f298a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c10cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38708/0 (socket says 172.21.15.59:38708) 2023-12-26T21:56:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.521+0000 7f298a59c700 1 -- 172.21.15.59:0/13870064 learned_addr learned my addr 172.21.15.59:0/13870064 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f298a59c700 1 -- 172.21.15.59:0/13870064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f297c00b200 con 0x7f298c0fc4d0 2023-12-26T21:56:18.532 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f298a59c700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c10cf20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f297c000c00 tx=0x7f297c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:18.533 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c01b410 con 0x7f298c0fc4d0 2023-12-26T21:56:18.533 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f298c10d610 con 0x7f298c0fc4d0 2023-12-26T21:56:18.533 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f298c10daa0 con 0x7f298c0fc4d0 2023-12-26T21:56:18.533 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f297c01b9d0 con 0x7f298c0fc4d0 2023-12-26T21:56:18.533 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.522+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f297c01a620 con 0x7f298c0fc4d0 2023-12-26T21:56:18.534 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.523+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2970004fa0 con 0x7f298c0fc4d0 2023-12-26T21:56:18.534 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.524+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f297c02b070 con 0x7f298c0fc4d0 2023-12-26T21:56:18.534 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.524+0000 7f2982ffd700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 0x7f296c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:18.534 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.524+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f297c01be40 con 0x7f298c0fc4d0 2023-12-26T21:56:18.534 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.524+0000 7f2989d9b700 1 -- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 msgr2=0x7f296c0334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:18.534 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.524+0000 7f2989d9b700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 0x7f296c0334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-26T21:56:18.535 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.528+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f297c023df0 con 0x7f298c0fc4d0 2023-12-26T21:56:18.724 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.725+0000 7f2989d9b700 1 -- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 msgr2=0x7f296c0334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:18.725 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.725+0000 7f2989d9b700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 0x7f296c0334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.750+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f2970005e80 con 0x7f298c0fc4d0 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.751+0000 7f2982ffd700 1 -- 172.21.15.59:0/13870064 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f297c032080 con 0x7f298c0fc4d0 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 5, 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": true, 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "active_name": "smithi059.gwkiyc", 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-26T21:56:18.751 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 msgr2=0x7f296c0334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 0x7f296c0334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 msgr2=0x7f298c10cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c10cf20 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f297c000c00 tx=0x7f297c006ee0).stop 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 shutdown_connections 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f296c030ff0 0x7f296c0334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 --2- 172.21.15.59:0/13870064 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f298c0fc4d0 0x7f298c10cf20 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.752+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 >> 172.21.15.59:0/13870064 conn(0x7f298c0f5e20 msgr2=0x7f298c0fa9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.753+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 shutdown_connections 2023-12-26T21:56:18.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:18.753+0000 7f2990b64700 1 -- 172.21.15.59:0/13870064 wait complete. 2023-12-26T21:56:19.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:18.751223+0000 mon.smithi059 (mon.0) 29 : audit [DBG] from='client.? 172.21.15.59:0/13870064' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-26T21:56:19.121 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for the mgr to restart... 2023-12-26T21:56:19.121 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr epoch 5... 2023-12-26T21:56:19.487 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.480+0000 7f355f8aa700 1 Processor -- start 2023-12-26T21:56:19.487 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355f8aa700 1 -- start start 2023-12-26T21:56:19.487 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355f8aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f35580fec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:19.487 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355f8aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35580ff110 con 0x7f3558027630 2023-12-26T21:56:19.487 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355d646700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f35580fec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:19.487 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355d646700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f35580fec20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38722/0 (socket says 172.21.15.59:38722) 2023-12-26T21:56:19.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355d646700 1 -- 172.21.15.59:0/3590384371 learned_addr learned my addr 172.21.15.59:0/3590384371 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:19.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355d646700 1 -- 172.21.15.59:0/3590384371 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35580ff250 con 0x7f3558027630 2023-12-26T21:56:19.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.481+0000 7f355d646700 1 --2- 172.21.15.59:0/3590384371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f35580fec20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f355400b5f0 tx=0x7f3554006ee0).ready entity=mon.0 client_cookie=a0fd392c1d76bd0b server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:19.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f3547fff700 1 -- 172.21.15.59:0/3590384371 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f355400d600 con 0x7f3558027630 2023-12-26T21:56:19.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f3547fff700 1 -- 172.21.15.59:0/3590384371 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f355400dbc0 con 0x7f3558027630 2023-12-26T21:56:19.488 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f3547fff700 1 -- 172.21.15.59:0/3590384371 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3554011b90 con 0x7f3558027630 2023-12-26T21:56:19.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 -- 172.21.15.59:0/3590384371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 msgr2=0x7f35580fec20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:19.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 --2- 172.21.15.59:0/3590384371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f35580fec20 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f355400b5f0 tx=0x7f3554006ee0).stop 2023-12-26T21:56:19.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 -- 172.21.15.59:0/3590384371 shutdown_connections 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 --2- 172.21.15.59:0/3590384371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f35580fec20 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 -- 172.21.15.59:0/3590384371 >> 172.21.15.59:0/3590384371 conn(0x7f35580f6680 msgr2=0x7f35580f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 -- 172.21.15.59:0/3590384371 shutdown_connections 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.482+0000 7f355f8aa700 1 -- 172.21.15.59:0/3590384371 wait complete. 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.483+0000 7f355f8aa700 1 Processor -- start 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.483+0000 7f355f8aa700 1 -- start start 2023-12-26T21:56:19.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.483+0000 7f355f8aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f355810ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.483+0000 7f355f8aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355810b220 con 0x7f3558027630 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.483+0000 7f355d646700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f355810ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f355d646700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f355810ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:38734/0 (socket says 172.21.15.59:38734) 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f355d646700 1 -- 172.21.15.59:0/2403865030 learned_addr learned my addr 172.21.15.59:0/2403865030 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f355d646700 1 -- 172.21.15.59:0/2403865030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f355400b200 con 0x7f3558027630 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f355d646700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f355810ad30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f3554000c00 tx=0x7f3554006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:19.491 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f355401b410 con 0x7f3558027630 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f355401b9d0 con 0x7f3558027630 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f355f8aa700 1 -- 172.21.15.59:0/2403865030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355810b420 con 0x7f3558027630 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f355401a620 con 0x7f3558027630 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.484+0000 7f355f8aa700 1 -- 172.21.15.59:0/2403865030 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355810c030 con 0x7f3558027630 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.485+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 5) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f355402b070 con 0x7f3558027630 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.485+0000 7f3545ffb700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.485+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 --> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3530000d10 con 0x7f3540030ff0 2023-12-26T21:56:19.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.485+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f3554043a50 con 0x7f3558027630 2023-12-26T21:56:19.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.485+0000 7f355ce45700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 msgr2=0x7f35400334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:19.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.486+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-26T21:56:19.686 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.687+0000 7f355ce45700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 msgr2=0x7f35400334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:19.687 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:19.687+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-26T21:56:20.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:20.088+0000 7f355ce45700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 msgr2=0x7f35400334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:20.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:20.088+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-26T21:56:20.888 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:20.889+0000 7f355ce45700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 msgr2=0x7f35400334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:20.889 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:20.889+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-26T21:56:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:22.490+0000 7f355ce45700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 msgr2=0x7f35400334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3091798160 2023-12-26T21:56:22.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:22.490+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-26T21:56:22.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:22.733+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 6) v1 ==== 36945+0+0 (secure 0 0 0) 0x7f355400f070 con 0x7f3558027630 2023-12-26T21:56:22.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:22.733+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 msgr2=0x7f35400334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-26T21:56:22.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:22.733+0000 7f3545ffb700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:23.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:22.117062+0000 mon.smithi059 (mon.0) 30 : cluster [INF] Active manager daemon smithi059.gwkiyc restarted 2023-12-26T21:56:23.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:22.117494+0000 mon.smithi059 (mon.0) 31 : cluster [INF] Activating manager daemon smithi059.gwkiyc 2023-12-26T21:56:23.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:22.119922+0000 mon.smithi059 (mon.0) 32 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-12-26T21:56:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.739+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f3554024400 con 0x7f3558027630 2023-12-26T21:56:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.739+0000 7f3545ffb700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3540034850 0x7f3540036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:23.739 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.739+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f3530000d10 con 0x7f3540034850 2023-12-26T21:56:23.740 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.739+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3540034850 0x7f3540036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:23.740 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.740+0000 7f355ce45700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3540034850 0x7f3540036c40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3548006fd0 tx=0x7f3548006d70).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:23.740 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.741+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f3530000d10 con 0x7f3540034850 2023-12-26T21:56:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.747+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f3530002460 con 0x7f3540034850 2023-12-26T21:56:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.747+0000 7f3545ffb700 1 -- 172.21.15.59:0/2403865030 <== mgr.14116 v2:172.21.15.59:6800/3069705549 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f3530002460 con 0x7f3540034850 2023-12-26T21:56:23.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-12-26T21:56:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "initialized": true 2023-12-26T21:56:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:23.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.747+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3540034850 msgr2=0x7f3540036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.747+0000 7f353f7fe700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3540034850 0x7f3540036c40 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f3548006fd0 tx=0x7f3548006d70).stop 2023-12-26T21:56:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 msgr2=0x7f355810ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f355810ad30 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f3554000c00 tx=0x7f3554006ee0).stop 2023-12-26T21:56:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 shutdown_connections 2023-12-26T21:56:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3540034850 0x7f3540036c40 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:23.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3558027630 0x7f355810ad30 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 --2- 172.21.15.59:0/2403865030 >> [v2:172.21.15.59:6800/3091798160,v1:172.21.15.59:6801/3091798160] conn(0x7f3540030ff0 0x7f35400334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 >> 172.21.15.59:0/2403865030 conn(0x7f35580f6680 msgr2=0x7f35580f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 shutdown_connections 2023-12-26T21:56:23.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:23.748+0000 7f353f7fe700 1 -- 172.21.15.59:0/2403865030 wait complete. 2023-12-26T21:56:24.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:22.734735+0000 mon.smithi059 (mon.0) 33 : cluster [DBG] mgrmap e6: smithi059.gwkiyc(active, starting, since 0.617329s) 2023-12-26T21:56:24.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.735818+0000 mon.smithi059 (mon.0) 34 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-26T21:56:24.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.736520+0000 mon.smithi059 (mon.0) 35 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.gwkiyc", "id": "smithi059.gwkiyc"}]: dispatch 2023-12-26T21:56:24.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.737366+0000 mon.smithi059 (mon.0) 36 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-26T21:56:24.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.737639+0000 mon.smithi059 (mon.0) 37 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-26T21:56:24.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.737862+0000 mon.smithi059 (mon.0) 38 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:22.751723+0000 mon.smithi059 (mon.0) 39 : cluster [INF] Manager daemon smithi059.gwkiyc is now available 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.767369+0000 mon.smithi059 (mon.0) 40 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.770408+0000 mon.smithi059 (mon.0) 41 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.773175+0000 mon.smithi059 (mon.0) 42 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.871207+0000 mon.smithi059 (mon.0) 43 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.871985+0000 mon.smithi059 (mon.0) 44 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.873552+0000 mon.smithi059 (mon.0) 45 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.875246+0000 mon.smithi059 (mon.0) 46 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.885889+0000 mon.smithi059 (mon.0) 47 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T21:56:24.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:22.887053+0000 mon.smithi059 (mon.0) 48 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T21:56:24.096 INFO:teuthology.orchestra.run.smithi059.stderr:mgr epoch 5 is available 2023-12-26T21:56:24.096 INFO:teuthology.orchestra.run.smithi059.stderr:Setting orchestrator backend to cephadm... 2023-12-26T21:56:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.614+0000 7f46d7024700 1 Processor -- start 2023-12-26T21:56:24.617 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.614+0000 7f46d7024700 1 -- start start 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.614+0000 7f46d7024700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d00fc8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.614+0000 7f46d7024700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d00fcde0 con 0x7f46d00fc4d0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46d4dc0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d00fc8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46d4dc0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d00fc8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36478/0 (socket says 172.21.15.59:36478) 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46d4dc0700 1 -- 172.21.15.59:0/814285942 learned_addr learned my addr 172.21.15.59:0/814285942 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46d4dc0700 1 -- 172.21.15.59:0/814285942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46d00fcf20 con 0x7f46d00fc4d0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46d4dc0700 1 --2- 172.21.15.59:0/814285942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d00fc8f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f46c400b5f0 tx=0x7f46c4006ee0).ready entity=mon.0 client_cookie=9582c6c623872c23 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46cf7fe700 1 -- 172.21.15.59:0/814285942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c400d600 con 0x7f46d00fc4d0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46cf7fe700 1 -- 172.21.15.59:0/814285942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f46c400dbc0 con 0x7f46d00fc4d0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.615+0000 7f46cf7fe700 1 -- 172.21.15.59:0/814285942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c4011b90 con 0x7f46d00fc4d0 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 -- 172.21.15.59:0/814285942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 msgr2=0x7f46d00fc8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:24.618 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 --2- 172.21.15.59:0/814285942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d00fc8f0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f46c400b5f0 tx=0x7f46c4006ee0).stop 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 -- 172.21.15.59:0/814285942 shutdown_connections 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 --2- 172.21.15.59:0/814285942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d00fc8f0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 -- 172.21.15.59:0/814285942 >> 172.21.15.59:0/814285942 conn(0x7f46d00f5e00 msgr2=0x7f46d00f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 -- 172.21.15.59:0/814285942 shutdown_connections 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 -- 172.21.15.59:0/814285942 wait complete. 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 Processor -- start 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.616+0000 7f46d7024700 1 -- start start 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d7024700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d010ceb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d7024700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46d010d3a0 con 0x7f46d00fc4d0 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d4dc0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d010ceb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d4dc0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d010ceb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36494/0 (socket says 172.21.15.59:36494) 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d4dc0700 1 -- 172.21.15.59:0/2115962087 learned_addr learned my addr 172.21.15.59:0/2115962087 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:24.619 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d4dc0700 1 -- 172.21.15.59:0/2115962087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46c400b200 con 0x7f46d00fc4d0 2023-12-26T21:56:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d4dc0700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d010ceb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f46c4000c00 tx=0x7f46c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c401b410 con 0x7f46d00fc4d0 2023-12-26T21:56:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f46d010d5a0 con 0x7f46d00fc4d0 2023-12-26T21:56:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f46c401b9d0 con 0x7f46d00fc4d0 2023-12-26T21:56:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f46c401a600 con 0x7f46d00fc4d0 2023-12-26T21:56:24.620 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.617+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46d010da30 con 0x7f46d00fc4d0 2023-12-26T21:56:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.618+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 7) v1 ==== 37015+0+0 (secure 0 0 0) 0x7f46c402b070 con 0x7f46d00fc4d0 2023-12-26T21:56:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.619+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f46d0045bb0 con 0x7f46d00fc4d0 2023-12-26T21:56:24.621 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.619+0000 7f46cd7fa700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f46b4030f30 0x7f46b40333f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.619+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f46c4042a80 con 0x7f46d00fc4d0 2023-12-26T21:56:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.619+0000 7f46cffff700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f46b4030f30 0x7f46b40333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.619+0000 7f46cffff700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f46b4030f30 0x7f46b40333f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f46bc0096a0 tx=0x7f46bc006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:24.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.623+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f46c401be20 con 0x7f46d00fc4d0 2023-12-26T21:56:24.741 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.741+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f46c400f070 con 0x7f46d00fc4d0 2023-12-26T21:56:24.797 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.797+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f46d010e290 con 0x7f46b4030f30 2023-12-26T21:56:24.802 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.803+0000 7f46cd7fa700 1 -- 172.21.15.59:0/2115962087 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f46d010e290 con 0x7f46b4030f30 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.804+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f46b4030f30 msgr2=0x7f46b40333f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.804+0000 7f46d7024700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f46b4030f30 0x7f46b40333f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f46bc0096a0 tx=0x7f46bc006b40).stop 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.804+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 msgr2=0x7f46d010ceb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.804+0000 7f46d7024700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d010ceb0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f46c4000c00 tx=0x7f46c4006ee0).stop 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.805+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 shutdown_connections 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.805+0000 7f46d7024700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f46b4030f30 0x7f46b40333f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.805+0000 7f46d7024700 1 --2- 172.21.15.59:0/2115962087 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f46d00fc4d0 0x7f46d010ceb0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:24.805 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.805+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 >> 172.21.15.59:0/2115962087 conn(0x7f46d00f5e00 msgr2=0x7f46d00ffcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:24.806 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.805+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 shutdown_connections 2023-12-26T21:56:24.806 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:24.805+0000 7f46d7024700 1 -- 172.21.15.59:0/2115962087 wait complete. 2023-12-26T21:56:25.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:23.740726+0000 mgr.smithi059.gwkiyc (mgr.14116) 1 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-26T21:56:25.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:23.741299+0000 mon.smithi059 (mon.0) 49 : cluster [DBG] mgrmap e7: smithi059.gwkiyc(active, since 1.62389s) 2023-12-26T21:56:25.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:23.747587+0000 mgr.smithi059.gwkiyc (mgr.14116) 2 : audit [DBG] from='client.14120 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-26T21:56:25.455 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28cabd700 1 Processor -- start 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28cabd700 1 -- start start 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28cabd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa288101070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28cabd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2880fca60 con 0x7fa288070c50 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa288101070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa288101070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36508/0 (socket says 172.21.15.59:36508) 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.451+0000 7fa28659c700 1 -- 172.21.15.59:0/4091953259 learned_addr learned my addr 172.21.15.59:0/4091953259 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:25.456 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28659c700 1 -- 172.21.15.59:0/4091953259 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa288101560 con 0x7fa288070c50 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28659c700 1 --2- 172.21.15.59:0/4091953259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa288101070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa27800b5f0 tx=0x7fa278006ee0).ready entity=mon.0 client_cookie=cbdf3adba9a8b78f server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28559a700 1 -- 172.21.15.59:0/4091953259 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27800d600 con 0x7fa288070c50 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28559a700 1 -- 172.21.15.59:0/4091953259 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa27800dbc0 con 0x7fa288070c50 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28559a700 1 -- 172.21.15.59:0/4091953259 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa278011b90 con 0x7fa288070c50 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28cabd700 1 -- 172.21.15.59:0/4091953259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 msgr2=0x7fa288101070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.452+0000 7fa28cabd700 1 --2- 172.21.15.59:0/4091953259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa288101070 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7fa27800b5f0 tx=0x7fa278006ee0).stop 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 -- 172.21.15.59:0/4091953259 shutdown_connections 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 --2- 172.21.15.59:0/4091953259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa288101070 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:25.457 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 -- 172.21.15.59:0/4091953259 >> 172.21.15.59:0/4091953259 conn(0x7fa2880f6690 msgr2=0x7fa2880f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 -- 172.21.15.59:0/4091953259 shutdown_connections 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 -- 172.21.15.59:0/4091953259 wait complete. 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 Processor -- start 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 -- start start 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa28810cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.453+0000 7fa28cabd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa28810d410 con 0x7fa288070c50 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa28810cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:25.458 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa28810cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36510/0 (socket says 172.21.15.59:36510) 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28659c700 1 -- 172.21.15.59:0/1398634546 learned_addr learned my addr 172.21.15.59:0/1398634546 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28659c700 1 -- 172.21.15.59:0/1398634546 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa27800b200 con 0x7fa288070c50 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28659c700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa28810cf20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa278000c00 tx=0x7fa278006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27800de00 con 0x7fa288070c50 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa27801b870 con 0x7fa288070c50 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa27801a670 con 0x7fa288070c50 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa28810d610 con 0x7fa288070c50 2023-12-26T21:56:25.459 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.454+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa28810daa0 con 0x7fa288070c50 2023-12-26T21:56:25.460 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.455+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa27802b070 con 0x7fa288070c50 2023-12-26T21:56:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.455+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa288045bb0 con 0x7fa288070c50 2023-12-26T21:56:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.455+0000 7fa276ffd700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa26c030ff0 0x7fa26c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.455+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa278042cd0 con 0x7fa288070c50 2023-12-26T21:56:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.455+0000 7fa285d9b700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa26c030ff0 0x7fa26c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.456+0000 7fa285d9b700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa26c030ff0 0x7fa26c0334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa27c0096a0 tx=0x7fa27c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:25.461 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.459+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa27801b9e0 con 0x7fa288070c50 2023-12-26T21:56:25.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.622+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7fa28810e300 con 0x7fa26c030ff0 2023-12-26T21:56:25.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.623+0000 7fa276ffd700 1 -- 172.21.15.59:0/1398634546 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7fa28810e300 con 0x7fa26c030ff0 2023-12-26T21:56:25.622 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: value unchanged 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa26c030ff0 msgr2=0x7fa26c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa26c030ff0 0x7fa26c0334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fa27c0096a0 tx=0x7fa27c006b40).stop 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 msgr2=0x7fa28810cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa28810cf20 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fa278000c00 tx=0x7fa278006ee0).stop 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 shutdown_connections 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa26c030ff0 0x7fa26c0334b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 --2- 172.21.15.59:0/1398634546 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa288070c50 0x7fa28810cf20 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.624+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 >> 172.21.15.59:0/1398634546 conn(0x7fa2880f6690 msgr2=0x7fa2880f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.625+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 shutdown_connections 2023-12-26T21:56:25.625 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:25.625+0000 7fa28cabd700 1 -- 172.21.15.59:0/1398634546 wait complete. 2023-12-26T21:56:25.897 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:24.742764+0000 mon.smithi059 (mon.0) 50 : cluster [DBG] mgrmap e8: smithi059.gwkiyc(active, since 2s) 2023-12-26T21:56:25.897 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:24.798121+0000 mgr.smithi059.gwkiyc (mgr.14116) 3 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:25.897 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:24.802930+0000 mon.smithi059 (mon.0) 51 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:25.897 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:24.806529+0000 mon.smithi059 (mon.0) 52 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:25.962 INFO:teuthology.orchestra.run.smithi059.stderr:Generating ssh key... 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.367+0000 7fa7cb9df700 1 Processor -- start 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.367+0000 7fa7cb9df700 1 -- start start 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.367+0000 7fa7cb9df700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c40febf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.367+0000 7fa7cb9df700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c40ff0e0 con 0x7fa7c4070cd0 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7c977b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c40febf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7c977b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c40febf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36520/0 (socket says 172.21.15.59:36520) 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7c977b700 1 -- 172.21.15.59:0/3295474539 learned_addr learned my addr 172.21.15.59:0/3295474539 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7c977b700 1 -- 172.21.15.59:0/3295474539 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c40ff220 con 0x7fa7c4070cd0 2023-12-26T21:56:26.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7c977b700 1 --2- 172.21.15.59:0/3295474539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c40febf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa7c000b5f0 tx=0x7fa7c0006ee0).ready entity=mon.0 client_cookie=708da477041c2b18 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7b3fff700 1 -- 172.21.15.59:0/3295474539 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c000d600 con 0x7fa7c4070cd0 2023-12-26T21:56:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7b3fff700 1 -- 172.21.15.59:0/3295474539 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa7c000dbc0 con 0x7fa7c4070cd0 2023-12-26T21:56:26.369 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.368+0000 7fa7b3fff700 1 -- 172.21.15.59:0/3295474539 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c0011b90 con 0x7fa7c4070cd0 2023-12-26T21:56:26.370 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 -- 172.21.15.59:0/3295474539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 msgr2=0x7fa7c40febf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 --2- 172.21.15.59:0/3295474539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c40febf0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fa7c000b5f0 tx=0x7fa7c0006ee0).stop 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 -- 172.21.15.59:0/3295474539 shutdown_connections 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 --2- 172.21.15.59:0/3295474539 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c40febf0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 -- 172.21.15.59:0/3295474539 >> 172.21.15.59:0/3295474539 conn(0x7fa7c40f6690 msgr2=0x7fa7c40f8ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 -- 172.21.15.59:0/3295474539 shutdown_connections 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.369+0000 7fa7cb9df700 1 -- 172.21.15.59:0/3295474539 wait complete. 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7cb9df700 1 Processor -- start 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7cb9df700 1 -- start start 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7cb9df700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c410ad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7cb9df700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7c410b250 con 0x7fa7c4070cd0 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7c977b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c410ad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7c977b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c410ad60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36522/0 (socket says 172.21.15.59:36522) 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7c977b700 1 -- 172.21.15.59:0/2776678459 learned_addr learned my addr 172.21.15.59:0/2776678459 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:26.371 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7c977b700 1 -- 172.21.15.59:0/2776678459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7c000b200 con 0x7fa7c4070cd0 2023-12-26T21:56:26.372 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.370+0000 7fa7c977b700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c410ad60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa7c0000c00 tx=0x7fa7c0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:26.372 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.371+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c001b410 con 0x7fa7c4070cd0 2023-12-26T21:56:26.372 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.371+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7c410b450 con 0x7fa7c4070cd0 2023-12-26T21:56:26.372 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.371+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa7c001b9d0 con 0x7fa7c4070cd0 2023-12-26T21:56:26.372 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.371+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7c001a620 con 0x7fa7c4070cd0 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.371+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7c410c060 con 0x7fa7c4070cd0 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.372+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7c40591e0 con 0x7fa7c4070cd0 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.372+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa7c002b070 con 0x7fa7c4070cd0 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.372+0000 7fa7b1ffb700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa7ac030fa0 0x7fa7ac033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.372+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa7c001be40 con 0x7fa7c4070cd0 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.373+0000 7fa7c8f7a700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa7ac030fa0 0x7fa7ac033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:26.373 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.373+0000 7fa7c8f7a700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa7ac030fa0 0x7fa7ac033460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa7b40096a0 tx=0x7fa7b4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:26.376 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.377+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa7c002c350 con 0x7fa7c4070cd0 2023-12-26T21:56:26.538 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:26.538+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa7c410c360 con 0x7fa7ac030fa0 2023-12-26T21:56:27.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:25.623201+0000 mgr.smithi059.gwkiyc (mgr.14116) 4 : audit [DBG] from='client.14128 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:27.261 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.260+0000 7fa7b1ffb700 1 -- 172.21.15.59:0/2776678459 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa7c410c360 con 0x7fa7ac030fa0 2023-12-26T21:56:27.262 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa7ac030fa0 msgr2=0x7fa7ac033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa7ac030fa0 0x7fa7ac033460 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fa7b40096a0 tx=0x7fa7b4006b40).stop 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 msgr2=0x7fa7c410ad60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c410ad60 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa7c0000c00 tx=0x7fa7c0006ee0).stop 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 shutdown_connections 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa7ac030fa0 0x7fa7ac033460 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 --2- 172.21.15.59:0/2776678459 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa7c4070cd0 0x7fa7c410ad60 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.262+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 >> 172.21.15.59:0/2776678459 conn(0x7fa7c40f6690 msgr2=0x7fa7c40f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.263+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 shutdown_connections 2023-12-26T21:56:27.264 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:27.263+0000 7fa7cb9df700 1 -- 172.21.15.59:0/2776678459 wait complete. 2023-12-26T21:56:28.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.024+0000 7fa3a1f60700 1 Processor -- start 2023-12-26T21:56:28.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.024+0000 7fa3a1f60700 1 -- start start 2023-12-26T21:56:28.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.024+0000 7fa3a1f60700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c0fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:28.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.024+0000 7fa3a1f60700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c0fb7e0 con 0x7fa39c0faed0 2023-12-26T21:56:28.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c0fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:28.030 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c0fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36530/0 (socket says 172.21.15.59:36530) 2023-12-26T21:56:28.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39b7fe700 1 -- 172.21.15.59:0/1014819969 learned_addr learned my addr 172.21.15.59:0/1014819969 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:28.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39b7fe700 1 -- 172.21.15.59:0/1014819969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39c0fb920 con 0x7fa39c0faed0 2023-12-26T21:56:28.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39b7fe700 1 --2- 172.21.15.59:0/1014819969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c0fb2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa39000b5f0 tx=0x7fa390006ee0).ready entity=mon.0 client_cookie=e8ee00711f09b4ba server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:28.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39a7fc700 1 -- 172.21.15.59:0/1014819969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa39000d600 con 0x7fa39c0faed0 2023-12-26T21:56:28.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39a7fc700 1 -- 172.21.15.59:0/1014819969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa39000dbc0 con 0x7fa39c0faed0 2023-12-26T21:56:28.031 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.025+0000 7fa39a7fc700 1 -- 172.21.15.59:0/1014819969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa390011b90 con 0x7fa39c0faed0 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 -- 172.21.15.59:0/1014819969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 msgr2=0x7fa39c0fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 --2- 172.21.15.59:0/1014819969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c0fb2f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa39000b5f0 tx=0x7fa390006ee0).stop 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 -- 172.21.15.59:0/1014819969 shutdown_connections 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 --2- 172.21.15.59:0/1014819969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c0fb2f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 -- 172.21.15.59:0/1014819969 >> 172.21.15.59:0/1014819969 conn(0x7fa39c0f6680 msgr2=0x7fa39c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 -- 172.21.15.59:0/1014819969 shutdown_connections 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 -- 172.21.15.59:0/1014819969 wait complete. 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.026+0000 7fa3a1f60700 1 Processor -- start 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa3a1f60700 1 -- start start 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa3a1f60700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c109be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa3a1f60700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa39c10a0d0 con 0x7fa39c0faed0 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa39b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c109be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:28.032 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa39b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c109be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36538/0 (socket says 172.21.15.59:36538) 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa39b7fe700 1 -- 172.21.15.59:0/2914052368 learned_addr learned my addr 172.21.15.59:0/2914052368 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa39b7fe700 1 -- 172.21.15.59:0/2914052368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa39000b200 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa39b7fe700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c109be0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa390000c00 tx=0x7fa390006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa39001b410 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fa39001b9d0 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa39c10a2d0 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa39001a620 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.027+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa39c10a760 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.028+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fa39002b070 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.028+0000 7fa383fff700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa388030ff0 0x7fa3880334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.028+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fa390042bf0 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.028+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa39c045bb0 con 0x7fa39c0faed0 2023-12-26T21:56:28.033 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.028+0000 7fa39affd700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa388030ff0 0x7fa3880334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:28.034 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.029+0000 7fa39affd700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa388030ff0 0x7fa3880334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa38c0096a0 tx=0x7fa38c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:28.034 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.033+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa39001f080 con 0x7fa39c0faed0 2023-12-26T21:56:28.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:27 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:26.538987+0000 mgr.smithi059.gwkiyc (mgr.14116) 6 : cephadm [INF] Generating ssh key... 2023-12-26T21:56:28.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:27 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:27.258899+0000 mon.smithi059 (mon.0) 53 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:28.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:27 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:27.260769+0000 mon.smithi059 (mon.0) 54 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:28.194 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.193+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fa39c10bd10 con 0x7fa388030ff0 2023-12-26T21:56:28.194 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.194+0000 7fa383fff700 1 -- 172.21.15.59:0/2914052368 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fa39c10bd10 con 0x7fa388030ff0 2023-12-26T21:56:28.194 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7EYrusFf986BdTlXSk15ugrJYzMD/3jLkCmGI93rPTqmn+VPRiuijQIMBMUYVFDJoaAdd5nsOU7WVw30E+p/BJJLhj0/7iprjrRih0eEmoA+XYFQVDZGwtUUtHxg4MGRTSy5U46eKkVxQhNpEBGzcmP/1WqJvL8wkXfQdkmgCtaW8qeF9srlmW/brdExsXjtqCXt7qtCUfJ2fPoX82Kj9jAcGmWQXgz/4JkujNgFgSG1bx/DHc0rqINxdb1ybF2hDXPyMKPB7ikmVudSwO6e25i+x2+beivlI6WVd/J4y5rLpUfdVT2YeiWNaKI/Dviq9H+MNo0B2w7WAaLgRjjnBMPXtg7yYD3xvHqqEMsnlGRpAfsADScSdNDMDcdPn7jM7ycJ4oXpKLcgqppmDnTobb5MsYVMUEAYZW0iywleSuq669080NVfOmc1fBxQf2tM0YmZIzxRtVakfTuYdJVrdWr23mDdE238JVDyMgvlvk1vz+Hj5wokJMvYkfq1Xy0= ceph-761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:56:28.196 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa388030ff0 msgr2=0x7fa3880334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa388030ff0 0x7fa3880334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fa38c0096a0 tx=0x7fa38c006b40).stop 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 msgr2=0x7fa39c109be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c109be0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fa390000c00 tx=0x7fa390006ee0).stop 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 shutdown_connections 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fa388030ff0 0x7fa3880334b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 --2- 172.21.15.59:0/2914052368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa39c0faed0 0x7fa39c109be0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 >> 172.21.15.59:0/2914052368 conn(0x7fa39c0f6680 msgr2=0x7fa39c0f8120 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 shutdown_connections 2023-12-26T21:56:28.197 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.195+0000 7fa3a1f60700 1 -- 172.21.15.59:0/2914052368 wait complete. 2023-12-26T21:56:28.529 INFO:teuthology.orchestra.run.smithi059.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-12-26T21:56:28.529 INFO:teuthology.orchestra.run.smithi059.stderr:Adding key to root@localhost authorized_keys... 2023-12-26T21:56:28.529 INFO:teuthology.orchestra.run.smithi059.stderr:Adding host smithi059... 2023-12-26T21:56:28.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.913+0000 7f14c59f5700 1 Processor -- start 2023-12-26T21:56:28.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.913+0000 7f14c59f5700 1 -- start start 2023-12-26T21:56:28.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.913+0000 7f14c59f5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c0070140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:28.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.913+0000 7f14c59f5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c0072060 con 0x7f14c0071c40 2023-12-26T21:56:28.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.913+0000 7f14beffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c0070140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:28.916 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14beffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c0070140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36544/0 (socket says 172.21.15.59:36544) 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14beffd700 1 -- 172.21.15.59:0/770533213 learned_addr learned my addr 172.21.15.59:0/770533213 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14beffd700 1 -- 172.21.15.59:0/770533213 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c0070630 con 0x7f14c0071c40 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14beffd700 1 --2- 172.21.15.59:0/770533213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c0070140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f14b400b5f0 tx=0x7f14b4006ee0).ready entity=mon.0 client_cookie=3de601f4977db29d server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14bdffb700 1 -- 172.21.15.59:0/770533213 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b400d600 con 0x7f14c0071c40 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14bdffb700 1 -- 172.21.15.59:0/770533213 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f14b400dbc0 con 0x7f14c0071c40 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.914+0000 7f14bdffb700 1 -- 172.21.15.59:0/770533213 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b4011b90 con 0x7f14c0071c40 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 -- 172.21.15.59:0/770533213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 msgr2=0x7f14c0070140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 --2- 172.21.15.59:0/770533213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c0070140 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f14b400b5f0 tx=0x7f14b4006ee0).stop 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 -- 172.21.15.59:0/770533213 shutdown_connections 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 --2- 172.21.15.59:0/770533213 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c0070140 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 -- 172.21.15.59:0/770533213 >> 172.21.15.59:0/770533213 conn(0x7f14c00f7790 msgr2=0x7f14c00f9bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 -- 172.21.15.59:0/770533213 shutdown_connections 2023-12-26T21:56:28.917 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.915+0000 7f14c59f5700 1 -- 172.21.15.59:0/770533213 wait complete. 2023-12-26T21:56:28.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14c59f5700 1 Processor -- start 2023-12-26T21:56:28.918 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14c59f5700 1 -- start start 2023-12-26T21:56:28.919 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14c59f5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c010cf30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14c59f5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14c010d420 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14beffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c010cf30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14beffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c010cf30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36554/0 (socket says 172.21.15.59:36554) 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14beffd700 1 -- 172.21.15.59:0/723185161 learned_addr learned my addr 172.21.15.59:0/723185161 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14beffd700 1 -- 172.21.15.59:0/723185161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14b400b200 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.916+0000 7f14beffd700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c010cf30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f14b400c010 tx=0x7f14b4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.917+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b400de90 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.917+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f14b401b870 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.917+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14b401a670 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.917+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14c010d620 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.917+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14c010dab0 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.918+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14c0045bb0 con 0x7f14c0071c40 2023-12-26T21:56:28.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.918+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f14b400f070 con 0x7f14c0071c40 2023-12-26T21:56:28.921 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.918+0000 7f14a77fe700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f14ac030ff0 0x7f14ac0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:28.921 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.918+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f14b4042a60 con 0x7f14c0071c40 2023-12-26T21:56:28.921 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.918+0000 7f14be7fc700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f14ac030ff0 0x7f14ac0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:28.921 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.918+0000 7f14be7fc700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f14ac030ff0 0x7f14ac0334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f14b00096a0 tx=0x7f14b0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:28.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:28.922+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f14b4019910 con 0x7f14c0071c40 2023-12-26T21:56:29.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.086+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}) v1 -- 0x7f14c010e2a0 con 0x7f14ac030ff0 2023-12-26T21:56:29.185 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:28.195109+0000 mgr.smithi059.gwkiyc (mgr.14116) 7 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:29.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Added host 'smithi059' with addr '172.21.15.59' 2023-12-26T21:56:29.710 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.708+0000 7f14a77fe700 1 -- 172.21.15.59:0/723185161 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7f14c010e2a0 con 0x7f14ac030ff0 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.710+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f14ac030ff0 msgr2=0x7f14ac0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.710+0000 7f14c59f5700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f14ac030ff0 0x7f14ac0334b0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f14b00096a0 tx=0x7f14b0006b40).stop 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 msgr2=0x7f14c010cf30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c010cf30 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f14b400c010 tx=0x7f14b4006ee0).stop 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 shutdown_connections 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f14ac030ff0 0x7f14ac0334b0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 --2- 172.21.15.59:0/723185161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14c0071c40 0x7f14c010cf30 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 >> 172.21.15.59:0/723185161 conn(0x7f14c00f7790 msgr2=0x7f14c00f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 shutdown_connections 2023-12-26T21:56:29.712 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:29.711+0000 7f14c59f5700 1 -- 172.21.15.59:0/723185161 wait complete. 2023-12-26T21:56:30.008 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying mon service with default placement... 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b74ddf700 1 Processor -- start 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b74ddf700 1 -- start start 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b74ddf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b700fea90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b74ddf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b700fef80 con 0x7f0b7006ad80 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b700fea90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b700fea90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36566/0 (socket says 172.21.15.59:36566) 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.361+0000 7f0b6f7fe700 1 -- 172.21.15.59:0/2599537322 learned_addr learned my addr 172.21.15.59:0/2599537322 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b6f7fe700 1 -- 172.21.15.59:0/2599537322 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b700ff0c0 con 0x7f0b7006ad80 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b6f7fe700 1 --2- 172.21.15.59:0/2599537322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b700fea90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0b6000b5f0 tx=0x7f0b60006ee0).ready entity=mon.0 client_cookie=66b4c1aa2f5baabc server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b6e7fc700 1 -- 172.21.15.59:0/2599537322 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6000d600 con 0x7f0b7006ad80 2023-12-26T21:56:30.364 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b6e7fc700 1 -- 172.21.15.59:0/2599537322 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0b6000dbc0 con 0x7f0b7006ad80 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b6e7fc700 1 -- 172.21.15.59:0/2599537322 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b60011b90 con 0x7f0b7006ad80 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b74ddf700 1 -- 172.21.15.59:0/2599537322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 msgr2=0x7f0b700fea90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b74ddf700 1 --2- 172.21.15.59:0/2599537322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b700fea90 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f0b6000b5f0 tx=0x7f0b60006ee0).stop 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b74ddf700 1 -- 172.21.15.59:0/2599537322 shutdown_connections 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b74ddf700 1 --2- 172.21.15.59:0/2599537322 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b700fea90 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.362+0000 7f0b74ddf700 1 -- 172.21.15.59:0/2599537322 >> 172.21.15.59:0/2599537322 conn(0x7f0b700f64d0 msgr2=0x7f0b700f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.363+0000 7f0b74ddf700 1 -- 172.21.15.59:0/2599537322 shutdown_connections 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.363+0000 7f0b74ddf700 1 -- 172.21.15.59:0/2599537322 wait complete. 2023-12-26T21:56:30.365 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.363+0000 7f0b74ddf700 1 Processor -- start 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b74ddf700 1 -- start start 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b74ddf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b7010ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b74ddf700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b7010b020 con 0x7f0b7006ad80 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b7010ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b7010ab30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36578/0 (socket says 172.21.15.59:36578) 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b6f7fe700 1 -- 172.21.15.59:0/1032956545 learned_addr learned my addr 172.21.15.59:0/1032956545 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b6f7fe700 1 -- 172.21.15.59:0/1032956545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b6000b200 con 0x7f0b7006ad80 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b6f7fe700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b7010ab30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0b60006800 tx=0x7f0b60006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.364+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6000de50 con 0x7f0b7006ad80 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.365+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f0b6001b870 con 0x7f0b7006ad80 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.365+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b7010b220 con 0x7f0b7006ad80 2023-12-26T21:56:30.367 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.365+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0b6001a7a0 con 0x7f0b7006ad80 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.365+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b7010b6b0 con 0x7f0b7006ad80 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.365+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f0b6000f070 con 0x7f0b7006ad80 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.365+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b70045bb0 con 0x7f0b7006ad80 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.366+0000 7f0b57fff700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f0b5c030ff0 0x7f0b5c0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.366+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f0b60042be0 con 0x7f0b7006ad80 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.366+0000 7f0b6effd700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f0b5c030ff0 0x7f0b5c0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:30.368 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.366+0000 7f0b6effd700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f0b5c030ff0 0x7f0b5c0334b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0b640096a0 tx=0x7f0b64006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:30.370 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.370+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0b6002b8e0 con 0x7f0b7006ad80 2023-12-26T21:56:30.537 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.537+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f0b7010bfd0 con 0x7f0b5c030ff0 2023-12-26T21:56:30.541 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.541+0000 7f0b57fff700 1 -- 172.21.15.59:0/1032956545 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f0b7010bfd0 con 0x7f0b5c030ff0 2023-12-26T21:56:30.541 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled mon update... 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f0b5c030ff0 msgr2=0x7f0b5c0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f0b5c030ff0 0x7f0b5c0334b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0b640096a0 tx=0x7f0b64006b40).stop 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 msgr2=0x7f0b7010ab30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b7010ab30 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f0b60006800 tx=0x7f0b60006ee0).stop 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 shutdown_connections 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f0b5c030ff0 0x7f0b5c0334b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 --2- 172.21.15.59:0/1032956545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0b7006ad80 0x7f0b7010ab30 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 >> 172.21.15.59:0/1032956545 conn(0x7f0b700f64d0 msgr2=0x7f0b700f7fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:30.544 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 shutdown_connections 2023-12-26T21:56:30.545 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:30.543+0000 7f0b74ddf700 1 -- 172.21.15.59:0/1032956545 wait complete. 2023-12-26T21:56:30.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:29.088276+0000 mgr.smithi059.gwkiyc (mgr.14116) 8 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi059", "addr": "172.21.15.59", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:30.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:29.708983+0000 mon.smithi059 (mon.0) 55 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:30.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:29.709315+0000 mgr.smithi059.gwkiyc (mgr.14116) 9 : cephadm [INF] Added host smithi059 2023-12-26T21:56:30.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:29.709575+0000 mon.smithi059 (mon.0) 56 : audit [DBG] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:30.855 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:30.542076+0000 mon.smithi059 (mon.0) 57 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:30.878 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying mgr service with default placement... 2023-12-26T21:56:31.239 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.237+0000 7f5ae23aa700 1 Processor -- start 2023-12-26T21:56:31.239 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.237+0000 7f5ae23aa700 1 -- start start 2023-12-26T21:56:31.239 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.237+0000 7f5ae23aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.237+0000 7f5ae23aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc0ff0d0 con 0x7f5adc027670 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.238+0000 7f5adbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.238+0000 7f5adbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36586/0 (socket says 172.21.15.59:36586) 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.238+0000 7f5adbfff700 1 -- 172.21.15.59:0/1512175758 learned_addr learned my addr 172.21.15.59:0/1512175758 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.238+0000 7f5adbfff700 1 -- 172.21.15.59:0/1512175758 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5adc0ff210 con 0x7f5adc027670 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.238+0000 7f5adbfff700 1 --2- 172.21.15.59:0/1512175758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc0febe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f5acc00b5f0 tx=0x7f5acc006ee0).ready entity=mon.0 client_cookie=21fa0c5f469cd69c server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.238+0000 7f5adaffd700 1 -- 172.21.15.59:0/1512175758 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5acc00d600 con 0x7f5adc027670 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5adaffd700 1 -- 172.21.15.59:0/1512175758 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5acc00dbc0 con 0x7f5adc027670 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5adaffd700 1 -- 172.21.15.59:0/1512175758 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5acc011b90 con 0x7f5adc027670 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 -- 172.21.15.59:0/1512175758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 msgr2=0x7f5adc0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:31.240 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 --2- 172.21.15.59:0/1512175758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc0febe0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f5acc00b5f0 tx=0x7f5acc006ee0).stop 2023-12-26T21:56:31.242 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 -- 172.21.15.59:0/1512175758 shutdown_connections 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 --2- 172.21.15.59:0/1512175758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc0febe0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 -- 172.21.15.59:0/1512175758 >> 172.21.15.59:0/1512175758 conn(0x7f5adc0f6680 msgr2=0x7f5adc0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 -- 172.21.15.59:0/1512175758 shutdown_connections 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.239+0000 7f5ae23aa700 1 -- 172.21.15.59:0/1512175758 wait complete. 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5ae23aa700 1 Processor -- start 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5ae23aa700 1 -- start start 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5ae23aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc10ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5ae23aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5adc10b220 con 0x7f5adc027670 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5adbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc10ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5adbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc10ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36598/0 (socket says 172.21.15.59:36598) 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5adbfff700 1 -- 172.21.15.59:0/2755340437 learned_addr learned my addr 172.21.15.59:0/2755340437 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.240+0000 7f5adbfff700 1 -- 172.21.15.59:0/2755340437 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5acc00b200 con 0x7f5adc027670 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.241+0000 7f5adbfff700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc10ad30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5acc006800 tx=0x7f5acc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:31.243 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.241+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5acc00de50 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.241+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5adc10b420 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.241+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f5acc01b870 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.241+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5acc01a7a0 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.241+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5adc10c030 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.242+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f5acc00f070 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.242+0000 7f5ad8ff9700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f5ac8030fa0 0x7f5ac8033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.242+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f5acc042be0 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.242+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5adc0591e0 con 0x7f5adc027670 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.242+0000 7f5adb7fe700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f5ac8030fa0 0x7f5ac8033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:31.244 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.242+0000 7f5adb7fe700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f5ac8030fa0 0x7f5ac8033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5ad00096a0 tx=0x7f5ad0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:31.248 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.246+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5acc016030 con 0x7f5adc027670 2023-12-26T21:56:31.416 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.415+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f5adc0ffff0 con 0x7f5ac8030fa0 2023-12-26T21:56:31.419 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.419+0000 7f5ad8ff9700 1 -- 172.21.15.59:0/2755340437 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f5adc0ffff0 con 0x7f5ac8030fa0 2023-12-26T21:56:31.420 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f5ac8030fa0 msgr2=0x7f5ac8033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f5ac8030fa0 0x7f5ac8033460 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f5ad00096a0 tx=0x7f5ad0006b40).stop 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 msgr2=0x7f5adc10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc10ad30 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f5acc006800 tx=0x7f5acc006ee0).stop 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 shutdown_connections 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f5ac8030fa0 0x7f5ac8033460 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 --2- 172.21.15.59:0/2755340437 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5adc027670 0x7f5adc10ad30 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 >> 172.21.15.59:0/2755340437 conn(0x7f5adc0f6680 msgr2=0x7f5adc0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:31.422 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 shutdown_connections 2023-12-26T21:56:31.423 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:31.421+0000 7f5ae23aa700 1 -- 172.21.15.59:0/2755340437 wait complete. 2023-12-26T21:56:31.799 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying crash service with default placement... 2023-12-26T21:56:32.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:30.539017+0000 mgr.smithi059.gwkiyc (mgr.14116) 10 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:32.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:30.539774+0000 mgr.smithi059.gwkiyc (mgr.14116) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-12-26T21:56:32.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:31.420219+0000 mon.smithi059 (mon.0) 58 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:32.156 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.153+0000 7f612315b700 1 Processor -- start 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.153+0000 7f612315b700 1 -- start start 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.153+0000 7f612315b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c0fea30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.153+0000 7f612315b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c0fef20 con 0x7f611c06ade0 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.153+0000 7f6122159700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c0fea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6122159700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c0fea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35074/0 (socket says 172.21.15.59:35074) 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6122159700 1 -- 172.21.15.59:0/3479505936 learned_addr learned my addr 172.21.15.59:0/3479505936 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6122159700 1 -- 172.21.15.59:0/3479505936 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611c0ff060 con 0x7f611c06ade0 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6122159700 1 --2- 172.21.15.59:0/3479505936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c0fea30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f611800b5f0 tx=0x7f6118006ee0).ready entity=mon.0 client_cookie=ea61d5c0d0ad0db2 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6121157700 1 -- 172.21.15.59:0/3479505936 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611800d600 con 0x7f611c06ade0 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6121157700 1 -- 172.21.15.59:0/3479505936 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f611800dbc0 con 0x7f611c06ade0 2023-12-26T21:56:32.157 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.154+0000 7f6121157700 1 -- 172.21.15.59:0/3479505936 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6118011b90 con 0x7f611c06ade0 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 -- 172.21.15.59:0/3479505936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 msgr2=0x7f611c0fea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 --2- 172.21.15.59:0/3479505936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c0fea30 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f611800b5f0 tx=0x7f6118006ee0).stop 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 -- 172.21.15.59:0/3479505936 shutdown_connections 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 --2- 172.21.15.59:0/3479505936 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c0fea30 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 -- 172.21.15.59:0/3479505936 >> 172.21.15.59:0/3479505936 conn(0x7f611c0f64d0 msgr2=0x7f611c0f88f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 -- 172.21.15.59:0/3479505936 shutdown_connections 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 -- 172.21.15.59:0/3479505936 wait complete. 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 Processor -- start 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.155+0000 7f612315b700 1 -- start start 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f612315b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c10ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f612315b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f611c10b010 con 0x7f611c06ade0 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f6122159700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c10ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:32.158 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f6122159700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c10ab20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35082/0 (socket says 172.21.15.59:35082) 2023-12-26T21:56:32.159 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f6122159700 1 -- 172.21.15.59:0/632986800 learned_addr learned my addr 172.21.15.59:0/632986800 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:32.159 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f6122159700 1 -- 172.21.15.59:0/632986800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f611800b200 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f6122159700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c10ab20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6118000c00 tx=0x7f6118006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611801b410 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f611801b9d0 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f611c10b210 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f611801a620 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.156+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f611c10b6a0 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.157+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f611802b070 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.157+0000 7f610affd700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f6104030fa0 0x7f6104033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.157+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f6118042c90 con 0x7f611c06ade0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.157+0000 7f6121958700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f6104030fa0 0x7f6104033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.158+0000 7f6121958700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f6104030fa0 0x7f6104033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f610c0096a0 tx=0x7f610c006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:32.160 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.158+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f611c045bb0 con 0x7f611c06ade0 2023-12-26T21:56:32.163 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.162+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6118016080 con 0x7f611c06ade0 2023-12-26T21:56:32.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.330+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f611c104b00 con 0x7f6104030fa0 2023-12-26T21:56:32.335 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.334+0000 7f610affd700 1 -- 172.21.15.59:0/632986800 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f611c104b00 con 0x7f6104030fa0 2023-12-26T21:56:32.335 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled crash update... 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f6104030fa0 msgr2=0x7f6104033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f6104030fa0 0x7f6104033460 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f610c0096a0 tx=0x7f610c006b40).stop 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 msgr2=0x7f611c10ab20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c10ab20 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f6118000c00 tx=0x7f6118006ee0).stop 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 shutdown_connections 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f6104030fa0 0x7f6104033460 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 --2- 172.21.15.59:0/632986800 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f611c06ade0 0x7f611c10ab20 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:32.338 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 >> 172.21.15.59:0/632986800 conn(0x7f611c0f64d0 msgr2=0x7f611c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:32.339 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 shutdown_connections 2023-12-26T21:56:32.339 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:32.336+0000 7f612315b700 1 -- 172.21.15.59:0/632986800 wait complete. 2023-12-26T21:56:32.692 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying prometheus service with default placement... 2023-12-26T21:56:32.872 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:31.417543+0000 mgr.smithi059.gwkiyc (mgr.14116) 12 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:32.872 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:31.418053+0000 mgr.smithi059.gwkiyc (mgr.14116) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-12-26T21:56:32.873 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:32.335095+0000 mon.smithi059 (mon.0) 59 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:33.078 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e6c3e700 1 Processor -- start 2023-12-26T21:56:33.079 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e6c3e700 1 -- start start 2023-12-26T21:56:33.081 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e6c3e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e0070180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e6c3e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e0070670 con 0x7f44e0071c80 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e5c3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e0070180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e5c3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e0070180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35094/0 (socket says 172.21.15.59:35094) 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e5c3c700 1 -- 172.21.15.59:0/177855906 learned_addr learned my addr 172.21.15.59:0/177855906 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.078+0000 7f44e5c3c700 1 -- 172.21.15.59:0/177855906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e00707b0 con 0x7f44e0071c80 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e5c3c700 1 --2- 172.21.15.59:0/177855906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e0070180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f44dc00b5f0 tx=0x7f44dc006ee0).ready entity=mon.0 client_cookie=be3cc49a162eb47d server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:33.082 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e4c3a700 1 -- 172.21.15.59:0/177855906 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44dc00d600 con 0x7f44e0071c80 2023-12-26T21:56:33.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e4c3a700 1 -- 172.21.15.59:0/177855906 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f44dc00dbc0 con 0x7f44e0071c80 2023-12-26T21:56:33.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e4c3a700 1 -- 172.21.15.59:0/177855906 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44dc011b90 con 0x7f44e0071c80 2023-12-26T21:56:33.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e6c3e700 1 -- 172.21.15.59:0/177855906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 msgr2=0x7f44e0070180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:33.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e6c3e700 1 --2- 172.21.15.59:0/177855906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e0070180 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f44dc00b5f0 tx=0x7f44dc006ee0).stop 2023-12-26T21:56:33.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e6c3e700 1 -- 172.21.15.59:0/177855906 shutdown_connections 2023-12-26T21:56:33.083 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e6c3e700 1 --2- 172.21.15.59:0/177855906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e0070180 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e6c3e700 1 -- 172.21.15.59:0/177855906 >> 172.21.15.59:0/177855906 conn(0x7f44e00f7600 msgr2=0x7f44e00f9a20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.079+0000 7f44e6c3e700 1 -- 172.21.15.59:0/177855906 shutdown_connections 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.080+0000 7f44e6c3e700 1 -- 172.21.15.59:0/177855906 wait complete. 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.080+0000 7f44e6c3e700 1 Processor -- start 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.080+0000 7f44e6c3e700 1 -- start start 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e6c3e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e010cd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e6c3e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e010d280 con 0x7f44e0071c80 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e5c3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e010cd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e5c3c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e010cd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35106/0 (socket says 172.21.15.59:35106) 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e5c3c700 1 -- 172.21.15.59:0/2218827929 learned_addr learned my addr 172.21.15.59:0/2218827929 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:33.084 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e5c3c700 1 -- 172.21.15.59:0/2218827929 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44dc00b200 con 0x7f44e0071c80 2023-12-26T21:56:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e5c3c700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e010cd90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f44dc006800 tx=0x7f44dc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:33.086 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44dc01b410 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f44dc01b9d0 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e010d480 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44dc01a760 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.081+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e010e090 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.082+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f44dc02b070 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.083+0000 7f44d27fc700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f44c8030ff0 0x7f44c80334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.082+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e00591e0 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.083+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f44dc042cc0 con 0x7f44e0071c80 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.083+0000 7f44e543b700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f44c8030ff0 0x7f44c80334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:33.087 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.083+0000 7f44e543b700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f44c8030ff0 0x7f44c80334b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f44d40096a0 tx=0x7f44d4006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:33.088 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.087+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f44dc01b570 con 0x7f44e0071c80 2023-12-26T21:56:33.256 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.255+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f44e010e2d0 con 0x7f44c8030ff0 2023-12-26T21:56:33.259 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.259+0000 7f44d27fc700 1 -- 172.21.15.59:0/2218827929 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f44e010e2d0 con 0x7f44c8030ff0 2023-12-26T21:56:33.260 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-12-26T21:56:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f44c8030ff0 msgr2=0x7f44c80334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:33.261 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f44c8030ff0 0x7f44c80334b0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f44d40096a0 tx=0x7f44d4006b40).stop 2023-12-26T21:56:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 msgr2=0x7f44e010cd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e010cd90 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f44dc006800 tx=0x7f44dc006ee0).stop 2023-12-26T21:56:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 shutdown_connections 2023-12-26T21:56:33.262 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f44c8030ff0 0x7f44c80334b0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 --2- 172.21.15.59:0/2218827929 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f44e0071c80 0x7f44e010cd90 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 >> 172.21.15.59:0/2218827929 conn(0x7f44e00f7600 msgr2=0x7f44e00f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 shutdown_connections 2023-12-26T21:56:33.263 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.261+0000 7f44e6c3e700 1 -- 172.21.15.59:0/2218827929 wait complete. 2023-12-26T21:56:33.594 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying grafana service with default placement... 2023-12-26T21:56:33.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.943+0000 7f696effe700 1 Processor -- start 2023-12-26T21:56:33.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.943+0000 7f696effe700 1 -- start start 2023-12-26T21:56:33.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696effe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f69680fc9b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:33.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696effe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69680fcea0 con 0x7f69680fc5b0 2023-12-26T21:56:33.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696cd9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f69680fc9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:33.945 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696cd9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f69680fc9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35114/0 (socket says 172.21.15.59:35114) 2023-12-26T21:56:33.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696cd9a700 1 -- 172.21.15.59:0/559342372 learned_addr learned my addr 172.21.15.59:0/559342372 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:33.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696cd9a700 1 -- 172.21.15.59:0/559342372 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69680fcfe0 con 0x7f69680fc5b0 2023-12-26T21:56:33.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f696cd9a700 1 --2- 172.21.15.59:0/559342372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f69680fc9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f695c00b5f0 tx=0x7f695c006ee0).ready entity=mon.0 client_cookie=5ea4d4ebb49bb7e6 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:33.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f69677fe700 1 -- 172.21.15.59:0/559342372 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f695c00d600 con 0x7f69680fc5b0 2023-12-26T21:56:33.946 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f69677fe700 1 -- 172.21.15.59:0/559342372 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f695c00dbc0 con 0x7f69680fc5b0 2023-12-26T21:56:33.947 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.944+0000 7f69677fe700 1 -- 172.21.15.59:0/559342372 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f695c011b90 con 0x7f69680fc5b0 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 -- 172.21.15.59:0/559342372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 msgr2=0x7f69680fc9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 --2- 172.21.15.59:0/559342372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f69680fc9b0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f695c00b5f0 tx=0x7f695c006ee0).stop 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 -- 172.21.15.59:0/559342372 shutdown_connections 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 --2- 172.21.15.59:0/559342372 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f69680fc9b0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 -- 172.21.15.59:0/559342372 >> 172.21.15.59:0/559342372 conn(0x7f69680f5e20 msgr2=0x7f69680f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 -- 172.21.15.59:0/559342372 shutdown_connections 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.945+0000 7f696effe700 1 -- 172.21.15.59:0/559342372 wait complete. 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696effe700 1 Processor -- start 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696effe700 1 -- start start 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696effe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f696810cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696effe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f696810d450 con 0x7f69680fc5b0 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696cd9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f696810cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696cd9a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f696810cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35130/0 (socket says 172.21.15.59:35130) 2023-12-26T21:56:33.948 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696cd9a700 1 -- 172.21.15.59:0/1505780082 learned_addr learned my addr 172.21.15.59:0/1505780082 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696cd9a700 1 -- 172.21.15.59:0/1505780082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f695c00b200 con 0x7f69680fc5b0 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.946+0000 7f696cd9a700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f696810cf60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f695c000c00 tx=0x7f695c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.947+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f695c011560 con 0x7f69680fc5b0 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.947+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f696810d650 con 0x7f69680fc5b0 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.947+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f695c01b870 con 0x7f69680fc5b0 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.947+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f695c01a690 con 0x7f69680fc5b0 2023-12-26T21:56:33.949 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.947+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f696810dae0 con 0x7f69680fc5b0 2023-12-26T21:56:33.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.947+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f695c02b070 con 0x7f69680fc5b0 2023-12-26T21:56:33.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.948+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6968045bb0 con 0x7f69680fc5b0 2023-12-26T21:56:33.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.948+0000 7f69657fa700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f694c030fa0 0x7f694c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:33.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.948+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f695c042bc0 con 0x7f69680fc5b0 2023-12-26T21:56:33.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.948+0000 7f6967fff700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f694c030fa0 0x7f694c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:33.950 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.948+0000 7f6967fff700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f694c030fa0 0x7f694c033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f69540096a0 tx=0x7f6954006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:33.952 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:33.952+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f695c01b9e0 con 0x7f69680fc5b0 2023-12-26T21:56:34.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:33 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:32.332289+0000 mgr.smithi059.gwkiyc (mgr.14116) 14 : audit [DBG] from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:34.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:33 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:32.332900+0000 mgr.smithi059.gwkiyc (mgr.14116) 15 : cephadm [INF] Saving service crash spec with placement * 2023-12-26T21:56:34.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:33 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:33.260221+0000 mon.smithi059 (mon.0) 60 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:34.121 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.120+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f696810e400 con 0x7f694c030fa0 2023-12-26T21:56:34.125 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.124+0000 7f69657fa700 1 -- 172.21.15.59:0/1505780082 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f696810e400 con 0x7f694c030fa0 2023-12-26T21:56:34.125 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-12-26T21:56:34.128 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f694c030fa0 msgr2=0x7f694c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:34.128 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f694c030fa0 0x7f694c033460 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f69540096a0 tx=0x7f6954006b40).stop 2023-12-26T21:56:34.128 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 msgr2=0x7f696810cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:34.128 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f696810cf60 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f695c000c00 tx=0x7f695c006ee0).stop 2023-12-26T21:56:34.128 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 shutdown_connections 2023-12-26T21:56:34.129 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f694c030fa0 0x7f694c033460 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:34.129 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 --2- 172.21.15.59:0/1505780082 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f69680fc5b0 0x7f696810cf60 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:34.129 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 >> 172.21.15.59:0/1505780082 conn(0x7f69680f5e20 msgr2=0x7f69680f7910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:34.129 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 shutdown_connections 2023-12-26T21:56:34.129 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.126+0000 7f696effe700 1 -- 172.21.15.59:0/1505780082 wait complete. 2023-12-26T21:56:34.511 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying node-exporter service with default placement... 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.894+0000 7f3f737f6700 1 Processor -- start 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.894+0000 7f3f737f6700 1 -- start start 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.894+0000 7f3f737f6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.894+0000 7f3f737f6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c0ff0d0 con 0x7f3f6c027670 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.894+0000 7f3f71592700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f71592700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35134/0 (socket says 172.21.15.59:35134) 2023-12-26T21:56:34.898 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f71592700 1 -- 172.21.15.59:0/589227934 learned_addr learned my addr 172.21.15.59:0/589227934 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f71592700 1 -- 172.21.15.59:0/589227934 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f6c0ff210 con 0x7f3f6c027670 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f71592700 1 --2- 172.21.15.59:0/589227934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c0febe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3f5c00b5f0 tx=0x7f3f5c006ee0).ready entity=mon.0 client_cookie=693c9caa6fb23cfa server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f5bfff700 1 -- 172.21.15.59:0/589227934 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f5c00d600 con 0x7f3f6c027670 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f5bfff700 1 -- 172.21.15.59:0/589227934 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f3f5c00dbc0 con 0x7f3f6c027670 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.895+0000 7f3f5bfff700 1 -- 172.21.15.59:0/589227934 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f5c011b90 con 0x7f3f6c027670 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 -- 172.21.15.59:0/589227934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 msgr2=0x7f3f6c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 --2- 172.21.15.59:0/589227934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c0febe0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f3f5c00b5f0 tx=0x7f3f5c006ee0).stop 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 -- 172.21.15.59:0/589227934 shutdown_connections 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 --2- 172.21.15.59:0/589227934 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c0febe0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 -- 172.21.15.59:0/589227934 >> 172.21.15.59:0/589227934 conn(0x7f3f6c0f6680 msgr2=0x7f3f6c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 -- 172.21.15.59:0/589227934 shutdown_connections 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 -- 172.21.15.59:0/589227934 wait complete. 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.896+0000 7f3f737f6700 1 Processor -- start 2023-12-26T21:56:34.899 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f737f6700 1 -- start start 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f737f6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c10ad30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f737f6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f6c10b220 con 0x7f3f6c027670 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f71592700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c10ad30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f71592700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c10ad30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35144/0 (socket says 172.21.15.59:35144) 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f71592700 1 -- 172.21.15.59:0/3446121202 learned_addr learned my addr 172.21.15.59:0/3446121202 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f71592700 1 -- 172.21.15.59:0/3446121202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f5c00b200 con 0x7f3f6c027670 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f71592700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c10ad30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3f5c006800 tx=0x7f3f5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f5c00de50 con 0x7f3f6c027670 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f3f5c01b870 con 0x7f3f6c027670 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3f5c01a7a0 con 0x7f3f6c027670 2023-12-26T21:56:34.900 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.897+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f6c10b420 con 0x7f3f6c027670 2023-12-26T21:56:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.898+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f6c10c030 con 0x7f3f6c027670 2023-12-26T21:56:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.898+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f3f5c00f070 con 0x7f3f6c027670 2023-12-26T21:56:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.899+0000 7f3f59ffb700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3f54030ff0 0x7f3f540334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:34.901 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.899+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f6c0591e0 con 0x7f3f6c027670 2023-12-26T21:56:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.899+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f3f5c042bb0 con 0x7f3f6c027670 2023-12-26T21:56:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.899+0000 7f3f70d91700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3f54030ff0 0x7f3f540334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:34.902 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.899+0000 7f3f70d91700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3f54030ff0 0x7f3f540334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f3f680096a0 tx=0x7f3f68006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:34.903 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:34.903+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3f5c016030 con 0x7f3f6c027670 2023-12-26T21:56:35.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:33.257454+0000 mgr.smithi059.gwkiyc (mgr.14116) 16 : audit [DBG] from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:35.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:33.258025+0000 mgr.smithi059.gwkiyc (mgr.14116) 17 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-12-26T21:56:35.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:34.125169+0000 mon.smithi059 (mon.0) 61 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:35.071 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.070+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f3f6c10c330 con 0x7f3f54030ff0 2023-12-26T21:56:35.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.074+0000 7f3f59ffb700 1 -- 172.21.15.59:0/3446121202 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f3f6c10c330 con 0x7f3f54030ff0 2023-12-26T21:56:35.075 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-12-26T21:56:35.077 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3f54030ff0 msgr2=0x7f3f540334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:35.077 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3f54030ff0 0x7f3f540334b0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f3f680096a0 tx=0x7f3f68006b40).stop 2023-12-26T21:56:35.077 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 msgr2=0x7f3f6c10ad30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:35.077 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c10ad30 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f3f5c006800 tx=0x7f3f5c006ee0).stop 2023-12-26T21:56:35.077 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 shutdown_connections 2023-12-26T21:56:35.077 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f3f54030ff0 0x7f3f540334b0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:35.078 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 --2- 172.21.15.59:0/3446121202 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3f6c027670 0x7f3f6c10ad30 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:35.078 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 >> 172.21.15.59:0/3446121202 conn(0x7f3f6c0f6680 msgr2=0x7f3f6c0f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:35.078 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 shutdown_connections 2023-12-26T21:56:35.078 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.076+0000 7f3f737f6700 1 -- 172.21.15.59:0/3446121202 wait complete. 2023-12-26T21:56:35.461 INFO:teuthology.orchestra.run.smithi059.stderr:Deploying alertmanager service with default placement... 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.802+0000 7f84d5865700 1 Processor -- start 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.802+0000 7f84d5865700 1 -- start start 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.802+0000 7f84d5865700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d00fc9f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.802+0000 7f84d5865700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d00fcee0 con 0x7f84d00fc5f0 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84ceffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d00fc9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84ceffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d00fc9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35156/0 (socket says 172.21.15.59:35156) 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84ceffd700 1 -- 172.21.15.59:0/3173778695 learned_addr learned my addr 172.21.15.59:0/3173778695 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84ceffd700 1 -- 172.21.15.59:0/3173778695 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84d00fd020 con 0x7f84d00fc5f0 2023-12-26T21:56:35.808 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84ceffd700 1 --2- 172.21.15.59:0/3173778695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d00fc9f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f84c400b5f0 tx=0x7f84c4006ee0).ready entity=mon.0 client_cookie=36b87309b4301a52 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84cdffb700 1 -- 172.21.15.59:0/3173778695 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84c400d600 con 0x7f84d00fc5f0 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84cdffb700 1 -- 172.21.15.59:0/3173778695 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f84c400dbc0 con 0x7f84d00fc5f0 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.803+0000 7f84cdffb700 1 -- 172.21.15.59:0/3173778695 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84c4011b90 con 0x7f84d00fc5f0 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 -- 172.21.15.59:0/3173778695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 msgr2=0x7f84d00fc9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 --2- 172.21.15.59:0/3173778695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d00fc9f0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f84c400b5f0 tx=0x7f84c4006ee0).stop 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 -- 172.21.15.59:0/3173778695 shutdown_connections 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 --2- 172.21.15.59:0/3173778695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d00fc9f0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 -- 172.21.15.59:0/3173778695 >> 172.21.15.59:0/3173778695 conn(0x7f84d00f5e20 msgr2=0x7f84d00f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 -- 172.21.15.59:0/3173778695 shutdown_connections 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.804+0000 7f84d5865700 1 -- 172.21.15.59:0/3173778695 wait complete. 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84d5865700 1 Processor -- start 2023-12-26T21:56:35.809 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84d5865700 1 -- start start 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84d5865700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d010cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84d5865700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84d010d450 con 0x7f84d00fc5f0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84ceffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d010cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84ceffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d010cf60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35170/0 (socket says 172.21.15.59:35170) 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84ceffd700 1 -- 172.21.15.59:0/3935763250 learned_addr learned my addr 172.21.15.59:0/3935763250 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84ceffd700 1 -- 172.21.15.59:0/3935763250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84c400b200 con 0x7f84d00fc5f0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.805+0000 7f84ceffd700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d010cf60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f84c400e040 tx=0x7f84c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.806+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84c401b410 con 0x7f84d00fc5f0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.806+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f84c401b9d0 con 0x7f84d00fc5f0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.806+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84d010d650 con 0x7f84d00fc5f0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.806+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84c401a7b0 con 0x7f84d00fc5f0 2023-12-26T21:56:35.810 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.806+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84d010dae0 con 0x7f84d00fc5f0 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.806+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f84c402b070 con 0x7f84d00fc5f0 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.807+0000 7f84b77fe700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f84bc030ff0 0x7f84bc0334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.807+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f84c4042d60 con 0x7f84d00fc5f0 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.807+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84d0045bb0 con 0x7f84d00fc5f0 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.807+0000 7f84ce7fc700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f84bc030ff0 0x7f84bc0334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.807+0000 7f84ce7fc700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f84bc030ff0 0x7f84bc0334b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f84c00096a0 tx=0x7f84c0006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:35.811 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.811+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f84c401f070 con 0x7f84d00fc5f0 2023-12-26T21:56:35.980 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.979+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f84d010e340 con 0x7f84bc030ff0 2023-12-26T21:56:35.984 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.983+0000 7f84b77fe700 1 -- 172.21.15.59:0/3935763250 <== mgr.14116 v2:172.21.15.59:6800/3069705549 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f84d010e340 con 0x7f84bc030ff0 2023-12-26T21:56:35.984 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-12-26T21:56:35.986 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f84bc030ff0 msgr2=0x7f84bc0334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:35.986 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f84bc030ff0 0x7f84bc0334b0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f84c00096a0 tx=0x7f84c0006b40).stop 2023-12-26T21:56:35.986 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 msgr2=0x7f84d010cf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d010cf60 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f84c400e040 tx=0x7f84c4006ee0).stop 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 shutdown_connections 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f84bc030ff0 0x7f84bc0334b0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 --2- 172.21.15.59:0/3935763250 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84d00fc5f0 0x7f84d010cf60 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 >> 172.21.15.59:0/3935763250 conn(0x7f84d00f5e20 msgr2=0x7f84d00f78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 shutdown_connections 2023-12-26T21:56:35.987 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:35.985+0000 7f84d5865700 1 -- 172.21.15.59:0/3935763250 wait complete. 2023-12-26T21:56:36.004 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:35 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:34.122358+0000 mgr.smithi059.gwkiyc (mgr.14116) 18 : audit [DBG] from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:36.005 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:35 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:34.122854+0000 mgr.smithi059.gwkiyc (mgr.14116) 19 : cephadm [INF] Saving service grafana spec with placement count:1 2023-12-26T21:56:36.005 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:35 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:35.075548+0000 mon.smithi059 (mon.0) 62 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.687+0000 7f91a5015700 1 Processor -- start 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.687+0000 7f91a5015700 1 -- start start 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.687+0000 7f91a5015700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a0101080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.687+0000 7f91a5015700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a00fca70 con 0x7f91a00275d0 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a0101080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a0101080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35186/0 (socket says 172.21.15.59:35186) 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919ed9d700 1 -- 172.21.15.59:0/3456980480 learned_addr learned my addr 172.21.15.59:0/3456980480 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919ed9d700 1 -- 172.21.15.59:0/3456980480 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91a0101570 con 0x7f91a00275d0 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919ed9d700 1 --2- 172.21.15.59:0/3456980480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a0101080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f919400b5f0 tx=0x7f9194006ee0).ready entity=mon.0 client_cookie=4c7476893253b39f server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:36.690 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919dd9b700 1 -- 172.21.15.59:0/3456980480 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919400d600 con 0x7f91a00275d0 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919dd9b700 1 -- 172.21.15.59:0/3456980480 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f919400dbc0 con 0x7f91a00275d0 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.688+0000 7f919dd9b700 1 -- 172.21.15.59:0/3456980480 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9194011b90 con 0x7f91a00275d0 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 -- 172.21.15.59:0/3456980480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 msgr2=0x7f91a0101080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 --2- 172.21.15.59:0/3456980480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a0101080 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f919400b5f0 tx=0x7f9194006ee0).stop 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 -- 172.21.15.59:0/3456980480 shutdown_connections 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 --2- 172.21.15.59:0/3456980480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a0101080 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 -- 172.21.15.59:0/3456980480 >> 172.21.15.59:0/3456980480 conn(0x7f91a00f6680 msgr2=0x7f91a00f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 -- 172.21.15.59:0/3456980480 shutdown_connections 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 -- 172.21.15.59:0/3456980480 wait complete. 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 Processor -- start 2023-12-26T21:56:36.691 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.689+0000 7f91a5015700 1 -- start start 2023-12-26T21:56:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f91a5015700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a010cf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f91a5015700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91a010d430 con 0x7f91a00275d0 2023-12-26T21:56:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f919ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a010cf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:36.693 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f919ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a010cf40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35192/0 (socket says 172.21.15.59:35192) 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f919ed9d700 1 -- 172.21.15.59:0/2315547420 learned_addr learned my addr 172.21.15.59:0/2315547420 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f919ed9d700 1 -- 172.21.15.59:0/2315547420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f919400b200 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f919ed9d700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a010cf40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9194006800 tx=0x7f9194006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919400d9c0 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7f919401b870 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91a010d630 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.690+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f919401a740 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.691+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f91a010dac0 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.691+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f91a0045bb0 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.692+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7f919402b070 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.692+0000 7f91877fe700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f918c030fa0 0x7f918c033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.692+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f9194042c80 con 0x7f91a00275d0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.692+0000 7f919e59c700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f918c030fa0 0x7f918c033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:36.694 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.692+0000 7f919e59c700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f918c030fa0 0x7f918c033460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f91900096a0 tx=0x7f9190006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:36.696 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.696+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f919401bcc0 con 0x7f91a00275d0 2023-12-26T21:56:36.861 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.860+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f91a0102150 con 0x7f91a00275d0 2023-12-26T21:56:36.866 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.866+0000 7f91877fe700 1 -- 172.21.15.59:0/2315547420 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f919401f070 con 0x7f91a00275d0 2023-12-26T21:56:36.868 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f918c030fa0 msgr2=0x7f918c033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f918c030fa0 0x7f918c033460 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f91900096a0 tx=0x7f9190006b40).stop 2023-12-26T21:56:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 msgr2=0x7f91a010cf40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:36.869 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a010cf40 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f9194006800 tx=0x7f9194006ee0).stop 2023-12-26T21:56:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 shutdown_connections 2023-12-26T21:56:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f918c030fa0 0x7f918c033460 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 --2- 172.21.15.59:0/2315547420 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f91a00275d0 0x7f91a010cf40 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.868+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 >> 172.21.15.59:0/2315547420 conn(0x7f91a00f6680 msgr2=0x7f91a00f8180 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.869+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 shutdown_connections 2023-12-26T21:56:36.870 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:36.869+0000 7f91a5015700 1 -- 172.21.15.59:0/2315547420 wait complete. 2023-12-26T21:56:37.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:35.072706+0000 mgr.smithi059.gwkiyc (mgr.14116) 20 : audit [DBG] from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:37.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:35.073254+0000 mgr.smithi059.gwkiyc (mgr.14116) 21 : cephadm [INF] Saving service node-exporter spec with placement * 2023-12-26T21:56:37.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:35.984386+0000 mon.smithi059 (mon.0) 63 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde83118700 1 Processor -- start 2023-12-26T21:56:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde83118700 1 -- start start 2023-12-26T21:56:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde83118700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c0fc8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde83118700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0fcdc0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.651 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde80eb4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c0fc8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde80eb4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c0fc8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35202/0 (socket says 172.21.15.59:35202) 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.647+0000 7fde80eb4700 1 -- 172.21.15.59:0/1323749754 learned_addr learned my addr 172.21.15.59:0/1323749754 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde80eb4700 1 -- 172.21.15.59:0/1323749754 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7c0fcf00 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde80eb4700 1 --2- 172.21.15.59:0/1323749754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c0fc8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fde7000b5f0 tx=0x7fde70006ee0).ready entity=mon.0 client_cookie=bbcb35b9c53e8396 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde7b7fe700 1 -- 172.21.15.59:0/1323749754 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7000d600 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde7b7fe700 1 -- 172.21.15.59:0/1323749754 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fde7000dbc0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde7b7fe700 1 -- 172.21.15.59:0/1323749754 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde70011b90 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.652 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde83118700 1 -- 172.21.15.59:0/1323749754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 msgr2=0x7fde7c0fc8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.648+0000 7fde83118700 1 --2- 172.21.15.59:0/1323749754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c0fc8d0 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fde7000b5f0 tx=0x7fde70006ee0).stop 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.649+0000 7fde83118700 1 -- 172.21.15.59:0/1323749754 shutdown_connections 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.649+0000 7fde83118700 1 --2- 172.21.15.59:0/1323749754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c0fc8d0 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.649+0000 7fde83118700 1 -- 172.21.15.59:0/1323749754 >> 172.21.15.59:0/1323749754 conn(0x7fde7c0f5e00 msgr2=0x7fde7c0f8220 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.649+0000 7fde83118700 1 -- 172.21.15.59:0/1323749754 shutdown_connections 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.649+0000 7fde83118700 1 -- 172.21.15.59:0/1323749754 wait complete. 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.649+0000 7fde83118700 1 Processor -- start 2023-12-26T21:56:37.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.650+0000 7fde83118700 1 -- start start 2023-12-26T21:56:37.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.650+0000 7fde83118700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c101130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:37.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.650+0000 7fde83118700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde7c0ff5b0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.650+0000 7fde80eb4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c101130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:37.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde80eb4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c101130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35210/0 (socket says 172.21.15.59:35210) 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde80eb4700 1 -- 172.21.15.59:0/1858765932 learned_addr learned my addr 172.21.15.59:0/1858765932 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde80eb4700 1 -- 172.21.15.59:0/1858765932 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde7000b200 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde80eb4700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c101130 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fde70000c00 tx=0x7fde70006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7001b410 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde7c0ff750 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 997+0+0 (secure 0 0 0) 0x7fde7001b9d0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde7001a620 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.651+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde7c0ffbe0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.652+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fde7002b070 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.652+0000 7fde797fa700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fde60030fa0 0x7fde60033460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.652+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fde70043a00 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.652+0000 7fde7bfff700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fde60030fa0 0x7fde60033460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.653+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde7c0591e0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.653+0000 7fde7bfff700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fde60030fa0 0x7fde60033460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fde680096a0 tx=0x7fde68006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:37.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.657+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fde7001f080 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.831 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.830+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fde7c045bb0 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.836 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.836+0000 7fde797fa700 1 -- 172.21.15.59:0/1858765932 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v9)=0 v9) v1 ==== 130+0+0 (secure 0 0 0) 0x7fde70016020 con 0x7fde7c0fc4b0 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fde60030fa0 msgr2=0x7fde60033460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fde60030fa0 0x7fde60033460 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fde680096a0 tx=0x7fde68006b40).stop 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 msgr2=0x7fde7c101130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c101130 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fde70000c00 tx=0x7fde70006ee0).stop 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 shutdown_connections 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fde60030fa0 0x7fde60033460 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:37.839 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 --2- 172.21.15.59:0/1858765932 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fde7c0fc4b0 0x7fde7c101130 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:37.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 >> 172.21.15.59:0/1858765932 conn(0x7fde7c0f5e00 msgr2=0x7fde7c0f6a00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:37.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 shutdown_connections 2023-12-26T21:56:37.840 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:37.838+0000 7fde83118700 1 -- 172.21.15.59:0/1858765932 wait complete. 2023-12-26T21:56:38.012 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:37 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:35.981612+0000 mgr.smithi059.gwkiyc (mgr.14116) 22 : audit [DBG] from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:38.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:37 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:35.982117+0000 mgr.smithi059.gwkiyc (mgr.14116) 23 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-12-26T21:56:38.013 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:37 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:36.867546+0000 mon.smithi059 (mon.0) 64 : audit [INF] from='client.? 172.21.15.59:0/2315547420' entity='client.admin' 2023-12-26T21:56:38.386 INFO:teuthology.orchestra.run.smithi059.stderr:Enabling the dashboard module... 2023-12-26T21:56:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.722+0000 7fb744f16700 1 Processor -- start 2023-12-26T21:56:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.722+0000 7fb744f16700 1 -- start start 2023-12-26T21:56:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb744f16700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb7400fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb744f16700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7400fb7e0 con 0x7fb7400faed0 2023-12-26T21:56:38.727 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb73e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb7400fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb73e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb7400fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35224/0 (socket says 172.21.15.59:35224) 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb73e59c700 1 -- 172.21.15.59:0/228971725 learned_addr learned my addr 172.21.15.59:0/228971725 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb73e59c700 1 -- 172.21.15.59:0/228971725 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7400fb920 con 0x7fb7400faed0 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.723+0000 7fb73e59c700 1 --2- 172.21.15.59:0/228971725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb7400fb2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb73000b5f0 tx=0x7fb730006ee0).ready entity=mon.0 client_cookie=3b43362306224f34 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb73d59a700 1 -- 172.21.15.59:0/228971725 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb73000d600 con 0x7fb7400faed0 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb73d59a700 1 -- 172.21.15.59:0/228971725 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb73000dbf0 con 0x7fb7400faed0 2023-12-26T21:56:38.728 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb73d59a700 1 -- 172.21.15.59:0/228971725 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb730011bf0 con 0x7fb7400faed0 2023-12-26T21:56:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 -- 172.21.15.59:0/228971725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 msgr2=0x7fb7400fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 --2- 172.21.15.59:0/228971725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb7400fb2f0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fb73000b5f0 tx=0x7fb730006ee0).stop 2023-12-26T21:56:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 -- 172.21.15.59:0/228971725 shutdown_connections 2023-12-26T21:56:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 --2- 172.21.15.59:0/228971725 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb7400fb2f0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 -- 172.21.15.59:0/228971725 >> 172.21.15.59:0/228971725 conn(0x7fb7400f6680 msgr2=0x7fb7400f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:38.729 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 -- 172.21.15.59:0/228971725 shutdown_connections 2023-12-26T21:56:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.724+0000 7fb744f16700 1 -- 172.21.15.59:0/228971725 wait complete. 2023-12-26T21:56:38.730 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.725+0000 7fb744f16700 1 Processor -- start 2023-12-26T21:56:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb744f16700 1 -- start start 2023-12-26T21:56:38.732 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb744f16700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb74010b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb744f16700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb74006adc0 con 0x7fb7400faed0 2023-12-26T21:56:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb73e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb74010b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb73e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb74010b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35226/0 (socket says 172.21.15.59:35226) 2023-12-26T21:56:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb73e59c700 1 -- 172.21.15.59:0/3143278345 learned_addr learned my addr 172.21.15.59:0/3143278345 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb73e59c700 1 -- 172.21.15.59:0/3143278345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb73000b200 con 0x7fb7400faed0 2023-12-26T21:56:38.733 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb73e59c700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb74010b800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb730011de0 tx=0x7fb730006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.726+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb73001b410 con 0x7fb7400faed0 2023-12-26T21:56:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.727+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb74006afc0 con 0x7fb7400faed0 2023-12-26T21:56:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.727+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb73001ba00 con 0x7fb7400faed0 2023-12-26T21:56:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.727+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb73001a770 con 0x7fb7400faed0 2023-12-26T21:56:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.727+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb74010bf30 con 0x7fb7400faed0 2023-12-26T21:56:38.734 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.728+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 8) v1 ==== 37095+0+0 (secure 0 0 0) 0x7fb73002b070 con 0x7fb7400faed0 2023-12-26T21:56:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.728+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb740045bb0 con 0x7fb7400faed0 2023-12-26T21:56:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.728+0000 7fb72effd700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fb724030ff0 0x7fb7240334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.728+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fb730042c60 con 0x7fb7400faed0 2023-12-26T21:56:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.730+0000 7fb73dd9b700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fb724030ff0 0x7fb7240334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:38.735 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.730+0000 7fb73dd9b700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fb724030ff0 0x7fb7240334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb7340096a0 tx=0x7fb734006b40).ready entity=mgr.14116 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:38.736 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.732+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb73002b8f0 con 0x7fb7400faed0 2023-12-26T21:56:38.943 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:38.943+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fb740027ae0 con 0x7fb7400faed0 2023-12-26T21:56:39.164 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:37.837402+0000 mon.smithi059 (mon.0) 65 : audit [INF] from='client.? 172.21.15.59:0/1858765932' entity='client.admin' 2023-12-26T21:56:39.165 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:38.175799+0000 mon.smithi059 (mon.0) 66 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:39.165 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:38.410259+0000 mon.smithi059 (mon.0) 67 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:39.889 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.888+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fb73000f070 con 0x7fb7400faed0 2023-12-26T21:56:39.890 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.890+0000 7fb72effd700 1 -- 172.21.15.59:0/3143278345 <== mon.0 v2:172.21.15.59:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fb730023d30 con 0x7fb7400faed0 2023-12-26T21:56:39.893 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fb724030ff0 msgr2=0x7fb7240334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:39.894 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fb724030ff0 0x7fb7240334b0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fb7340096a0 tx=0x7fb734006b40).stop 2023-12-26T21:56:39.894 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 msgr2=0x7fb74010b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:39.894 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb74010b800 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fb730011de0 tx=0x7fb730006ee0).stop 2023-12-26T21:56:39.894 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 shutdown_connections 2023-12-26T21:56:39.895 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fb724030ff0 0x7fb7240334b0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:39.895 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 --2- 172.21.15.59:0/3143278345 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb7400faed0 0x7fb74010b800 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:39.895 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.893+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 >> 172.21.15.59:0/3143278345 conn(0x7fb7400f6680 msgr2=0x7fb7400f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:39.895 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.894+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 shutdown_connections 2023-12-26T21:56:39.895 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:39.894+0000 7fb744f16700 1 -- 172.21.15.59:0/3143278345 wait complete. 2023-12-26T21:56:40.150 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:39 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:38.888518+0000 mon.smithi059 (mon.0) 68 : audit [INF] from='mgr.14116 172.21.15.59:0/4078074858' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:40.151 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:39 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:38.944630+0000 mon.smithi059 (mon.0) 69 : audit [INF] from='client.? 172.21.15.59:0/3143278345' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-12-26T21:56:40.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f2cdd700 1 Processor -- start 2023-12-26T21:56:40.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f2cdd700 1 -- start start 2023-12-26T21:56:40.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f2cdd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec070160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:40.653 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f2cdd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec072080 con 0x7fd8ec071c60 2023-12-26T21:56:40.654 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f0a79700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec070160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:40.654 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f0a79700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec070160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35240/0 (socket says 172.21.15.59:35240) 2023-12-26T21:56:40.654 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f0a79700 1 -- 172.21.15.59:0/2800340470 learned_addr learned my addr 172.21.15.59:0/2800340470 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f0a79700 1 -- 172.21.15.59:0/2800340470 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8ec070650 con 0x7fd8ec071c60 2023-12-26T21:56:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.652+0000 7fd8f0a79700 1 --2- 172.21.15.59:0/2800340470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec070160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd8e000b5f0 tx=0x7fd8e0006ee0).ready entity=mon.0 client_cookie=9da9a7b563dec83b server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:40.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8eb7fe700 1 -- 172.21.15.59:0/2800340470 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e000d600 con 0x7fd8ec071c60 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8eb7fe700 1 -- 172.21.15.59:0/2800340470 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd8e000dbf0 con 0x7fd8ec071c60 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8eb7fe700 1 -- 172.21.15.59:0/2800340470 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e0011bf0 con 0x7fd8ec071c60 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/2800340470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 msgr2=0x7fd8ec070160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 --2- 172.21.15.59:0/2800340470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec070160 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fd8e000b5f0 tx=0x7fd8e0006ee0).stop 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/2800340470 shutdown_connections 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 --2- 172.21.15.59:0/2800340470 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec070160 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/2800340470 >> 172.21.15.59:0/2800340470 conn(0x7fd8ec0f7770 msgr2=0x7fd8ec0f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/2800340470 shutdown_connections 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.653+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/2800340470 wait complete. 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.654+0000 7fd8f2cdd700 1 Processor -- start 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.654+0000 7fd8f2cdd700 1 -- start start 2023-12-26T21:56:40.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.654+0000 7fd8f2cdd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.654+0000 7fd8f2cdd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8ec10d3f0 con 0x7fd8ec071c60 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f0a79700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f0a79700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35244/0 (socket says 172.21.15.59:35244) 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f0a79700 1 -- 172.21.15.59:0/3771112179 learned_addr learned my addr 172.21.15.59:0/3771112179 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f0a79700 1 -- 172.21.15.59:0/3771112179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8e000b200 con 0x7fd8ec071c60 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f0a79700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec10cf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd8e000bdc0 tx=0x7fd8e0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e001b410 con 0x7fd8ec071c60 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8ec10d5f0 con 0x7fd8ec071c60 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd8e001ba00 con 0x7fd8ec071c60 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8e001a6b0 con 0x7fd8ec071c60 2023-12-26T21:56:40.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.655+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8ec10da80 con 0x7fd8ec071c60 2023-12-26T21:56:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.656+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8ec045bb0 con 0x7fd8ec071c60 2023-12-26T21:56:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.656+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fd8e002b020 con 0x7fd8ec071c60 2023-12-26T21:56:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.657+0000 7fd8e97fa700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 0x7fd8d00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.657+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7fd8e0042800 con 0x7fd8ec071c60 2023-12-26T21:56:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.657+0000 7fd8ebfff700 1 -- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 msgr2=0x7fd8d00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:40.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.657+0000 7fd8ebfff700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 0x7fd8d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-26T21:56:40.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.661+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd8e00103a0 con 0x7fd8ec071c60 2023-12-26T21:56:40.858 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.858+0000 7fd8ebfff700 1 -- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 msgr2=0x7fd8d00334b0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:40.859 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.858+0000 7fd8ebfff700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 0x7fd8d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-26T21:56:40.864 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.863+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fd8ec1069d0 con 0x7fd8ec071c60 2023-12-26T21:56:40.864 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.863+0000 7fd8e97fa700 1 -- 172.21.15.59:0/3771112179 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fd8e001f080 con 0x7fd8ec071c60 2023-12-26T21:56:40.864 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:40.865 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "epoch": 9, 2023-12-26T21:56:40.865 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "available": true, 2023-12-26T21:56:40.865 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "active_name": "smithi059.gwkiyc", 2023-12-26T21:56:40.865 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "num_standby": 0 2023-12-26T21:56:40.865 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 msgr2=0x7fd8d00334b0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 0x7fd8d00334b0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 msgr2=0x7fd8ec10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec10cf00 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fd8e000bdc0 tx=0x7fd8e0006ee0).stop 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 shutdown_connections 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7fd8d0030ff0 0x7fd8d00334b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 --2- 172.21.15.59:0/3771112179 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8ec071c60 0x7fd8ec10cf00 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 >> 172.21.15.59:0/3771112179 conn(0x7fd8ec0f7770 msgr2=0x7fd8ec0f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:40.867 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 shutdown_connections 2023-12-26T21:56:40.868 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:40.865+0000 7fd8f2cdd700 1 -- 172.21.15.59:0/3771112179 wait complete. 2023-12-26T21:56:41.253 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for the mgr to restart... 2023-12-26T21:56:41.253 INFO:teuthology.orchestra.run.smithi059.stderr:Waiting for mgr epoch 9... 2023-12-26T21:56:41.256 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:39.891111+0000 mon.smithi059 (mon.0) 70 : audit [INF] from='client.? 172.21.15.59:0/3143278345' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-12-26T21:56:41.256 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:39.891194+0000 mon.smithi059 (mon.0) 71 : cluster [DBG] mgrmap e9: smithi059.gwkiyc(active, since 17s) 2023-12-26T21:56:41.256 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:40.864690+0000 mon.smithi059 (mon.0) 72 : audit [DBG] from='client.? 172.21.15.59:0/3771112179' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-12-26T21:56:41.655 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.652+0000 7f781e923700 1 Processor -- start 2023-12-26T21:56:41.655 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.652+0000 7f781e923700 1 -- start start 2023-12-26T21:56:41.655 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.652+0000 7f781e923700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f78180fc3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:41.655 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.652+0000 7f781e923700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78180fc8d0 con 0x7f78180fbfc0 2023-12-26T21:56:41.655 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7817fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f78180fc3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7817fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f78180fc3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35252/0 (socket says 172.21.15.59:35252) 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7817fff700 1 -- 172.21.15.59:0/1139780177 learned_addr learned my addr 172.21.15.59:0/1139780177 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7817fff700 1 -- 172.21.15.59:0/1139780177 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f78180fca10 con 0x7f78180fbfc0 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7817fff700 1 --2- 172.21.15.59:0/1139780177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f78180fc3e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f780800b5f0 tx=0x7f7808006ee0).ready entity=mon.0 client_cookie=9ade9b922df75a40 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7816ffd700 1 -- 172.21.15.59:0/1139780177 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f780800d600 con 0x7f78180fbfc0 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7816ffd700 1 -- 172.21.15.59:0/1139780177 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f780800dbf0 con 0x7f78180fbfc0 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.653+0000 7f7816ffd700 1 -- 172.21.15.59:0/1139780177 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7808011bf0 con 0x7f78180fbfc0 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 -- 172.21.15.59:0/1139780177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 msgr2=0x7f78180fc3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 --2- 172.21.15.59:0/1139780177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f78180fc3e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f780800b5f0 tx=0x7f7808006ee0).stop 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 -- 172.21.15.59:0/1139780177 shutdown_connections 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 --2- 172.21.15.59:0/1139780177 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f78180fc3e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 -- 172.21.15.59:0/1139780177 >> 172.21.15.59:0/1139780177 conn(0x7f78180f7770 msgr2=0x7f78180f9b90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 -- 172.21.15.59:0/1139780177 shutdown_connections 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 -- 172.21.15.59:0/1139780177 wait complete. 2023-12-26T21:56:41.656 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.654+0000 7f781e923700 1 Processor -- start 2023-12-26T21:56:41.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f781e923700 1 -- start start 2023-12-26T21:56:41.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f781e923700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f7818071c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:41.657 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f781e923700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f78180700e0 con 0x7f78180fbfc0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f7817fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f7818071c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f7817fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f7818071c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:35266/0 (socket says 172.21.15.59:35266) 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f7817fff700 1 -- 172.21.15.59:0/681920078 learned_addr learned my addr 172.21.15.59:0/681920078 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f7817fff700 1 -- 172.21.15.59:0/681920078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f780800b200 con 0x7f78180fbfc0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f7817fff700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f7818071c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7808006800 tx=0x7f7808006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.655+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f780800dec0 con 0x7f78180fbfc0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.656+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f780801b8a0 con 0x7f78180fbfc0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.656+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7818070280 con 0x7f78180fbfc0 2023-12-26T21:56:41.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.656+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f780801a7b0 con 0x7f78180fbfc0 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.656+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7818070710 con 0x7f78180fbfc0 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.656+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 9) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f780802b070 con 0x7f78180fbfc0 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.657+0000 7f7814ff9700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.657+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 --> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7804033bc0 con 0x7f7804031040 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.657+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 893+0+0 (secure 0 0 0) 0x7f7808042c40 con 0x7f78180fbfc0 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.657+0000 7f78177fe700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 msgr2=0x7f7804033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:41.659 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.657+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-12-26T21:56:41.858 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.858+0000 7f78177fe700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 msgr2=0x7f7804033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:41.859 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:41.858+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-12-26T21:56:42.259 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:42.259+0000 7f78177fe700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 msgr2=0x7f7804033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:42.260 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:42.259+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-12-26T21:56:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:43.060+0000 7f78177fe700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 msgr2=0x7f7804033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:43.060 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:43.060+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-12-26T21:56:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:44.661+0000 7f78177fe700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 msgr2=0x7f7804033500 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.59:6800/3069705549 2023-12-26T21:56:44.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:44.661+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-12-26T21:56:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:45.317+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mgrmap(e 10) v1 ==== 36958+0+0 (secure 0 0 0) 0x7f780800f070 con 0x7f78180fbfc0 2023-12-26T21:56:45.317 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:45.317+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 msgr2=0x7f7804033500 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-12-26T21:56:45.318 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:45.317+0000 7f7814ff9700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:45.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:45.263293+0000 mon.smithi059 (mon.0) 73 : cluster [INF] Active manager daemon smithi059.gwkiyc restarted 2023-12-26T21:56:45.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:45.263689+0000 mon.smithi059 (mon.0) 74 : cluster [INF] Activating manager daemon smithi059.gwkiyc 2023-12-26T21:56:45.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:45.267722+0000 mon.smithi059 (mon.0) 75 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-12-26T21:56:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.320+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f78080440d0 con 0x7f78180fbfc0 2023-12-26T21:56:46.322 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.320+0000 7f7814ff9700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7804034b90 0x7f7804036f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:46.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.320+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7804033bc0 con 0x7f7804034b90 2023-12-26T21:56:46.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.321+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7804034b90 0x7f7804036f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:46.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.321+0000 7f78177fe700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7804034b90 0x7f7804036f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f780c006fd0 tx=0x7f780c006db0).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:46.323 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.322+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f7804033bc0 con 0x7f7804034b90 2023-12-26T21:56:46.331 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.328+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f78180591e0 con 0x7f7804034b90 2023-12-26T21:56:46.331 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f7814ff9700 1 -- 172.21.15.59:0/681920078 <== mgr.14156 v2:172.21.15.59:6800/1607822224 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7f78180591e0 con 0x7f7804034b90 2023-12-26T21:56:46.331 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: { 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "mgrmap_epoch": 11, 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: "initialized": true 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: } 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7804034b90 msgr2=0x7f7804036f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7804034b90 0x7f7804036f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f780c006fd0 tx=0x7f780c006db0).stop 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 msgr2=0x7f7818071c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f7818071c60 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f7808006800 tx=0x7f7808006ee0).stop 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 shutdown_connections 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7804034b90 0x7f7804036f80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:46.332 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f78180fbfc0 0x7f7818071c60 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:46.333 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 --2- 172.21.15.59:0/681920078 >> [v2:172.21.15.59:6800/3069705549,v1:172.21.15.59:6801/3069705549] conn(0x7f7804031040 0x7f7804033500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-12-26T21:56:46.333 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 >> 172.21.15.59:0/681920078 conn(0x7f78180f7770 msgr2=0x7f78180f9210 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:46.333 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 shutdown_connections 2023-12-26T21:56:46.333 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:46.329+0000 7f781e923700 1 -- 172.21.15.59:0/681920078 wait complete. 2023-12-26T21:56:46.766 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:45.318941+0000 mon.smithi059 (mon.0) 76 : cluster [DBG] mgrmap e10: smithi059.gwkiyc(active, starting, since 0.0553373s) 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.321535+0000 mon.smithi059 (mon.0) 77 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mon metadata", "id": "smithi059"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.322152+0000 mon.smithi059 (mon.0) 78 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mgr metadata", "who": "smithi059.gwkiyc", "id": "smithi059.gwkiyc"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.323050+0000 mon.smithi059 (mon.0) 79 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.323249+0000 mon.smithi059 (mon.0) 80 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.323396+0000 mon.smithi059 (mon.0) 81 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:45.336629+0000 mon.smithi059 (mon.0) 82 : cluster [INF] Manager daemon smithi059.gwkiyc is now available 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.441836+0000 mon.smithi059 (mon.0) 83 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.444248+0000 mon.smithi059 (mon.0) 84 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.463702+0000 mon.smithi059 (mon.0) 85 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:45.465196+0000 mon.smithi059 (mon.0) 86 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T21:56:46.767 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:46.067844+0000 mon.smithi059 (mon.0) 87 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:47.076 INFO:teuthology.orchestra.run.smithi059.stderr:mgr epoch 9 is available 2023-12-26T21:56:47.076 INFO:teuthology.orchestra.run.smithi059.stderr:Generating a dashboard self-signed certificate... 2023-12-26T21:56:47.468 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8d06648700 1 Processor -- start 2023-12-26T21:56:47.468 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8d06648700 1 -- start start 2023-12-26T21:56:47.468 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8d06648700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d000fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:47.469 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8d06648700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d000fb7e0 con 0x7f8d000faed0 2023-12-26T21:56:47.469 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d000fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:47.469 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d000fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36546/0 (socket says 172.21.15.59:36546) 2023-12-26T21:56:47.469 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.460+0000 7f8cfffff700 1 -- 172.21.15.59:0/626851807 learned_addr learned my addr 172.21.15.59:0/626851807 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:47.469 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8cfffff700 1 -- 172.21.15.59:0/626851807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d000fb920 con 0x7f8d000faed0 2023-12-26T21:56:47.469 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8cfffff700 1 --2- 172.21.15.59:0/626851807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d000fb2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8cf000b5f0 tx=0x7f8cf0006ee0).ready entity=mon.0 client_cookie=56f2d1b96383ebe8 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8cfeffd700 1 -- 172.21.15.59:0/626851807 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cf000d600 con 0x7f8d000faed0 2023-12-26T21:56:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8cfeffd700 1 -- 172.21.15.59:0/626851807 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8cf000dbf0 con 0x7f8d000faed0 2023-12-26T21:56:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8cfeffd700 1 -- 172.21.15.59:0/626851807 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cf0011bf0 con 0x7f8d000faed0 2023-12-26T21:56:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8d06648700 1 -- 172.21.15.59:0/626851807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 msgr2=0x7f8d000fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:47.470 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.461+0000 7f8d06648700 1 --2- 172.21.15.59:0/626851807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d000fb2f0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f8cf000b5f0 tx=0x7f8cf0006ee0).stop 2023-12-26T21:56:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 -- 172.21.15.59:0/626851807 shutdown_connections 2023-12-26T21:56:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 --2- 172.21.15.59:0/626851807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d000fb2f0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 -- 172.21.15.59:0/626851807 >> 172.21.15.59:0/626851807 conn(0x7f8d000f6680 msgr2=0x7f8d000f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 -- 172.21.15.59:0/626851807 shutdown_connections 2023-12-26T21:56:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 -- 172.21.15.59:0/626851807 wait complete. 2023-12-26T21:56:47.471 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 Processor -- start 2023-12-26T21:56:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.462+0000 7f8d06648700 1 -- start start 2023-12-26T21:56:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8d06648700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d0010cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8d06648700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0010d410 con 0x7f8d000faed0 2023-12-26T21:56:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d0010cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d0010cf20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36560/0 (socket says 172.21.15.59:36560) 2023-12-26T21:56:47.472 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfffff700 1 -- 172.21.15.59:0/564130968 learned_addr learned my addr 172.21.15.59:0/564130968 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfffff700 1 -- 172.21.15.59:0/564130968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf000b200 con 0x7f8d000faed0 2023-12-26T21:56:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfffff700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d0010cf20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8cf0011de0 tx=0x7f8cf0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cf001b410 con 0x7f8d000faed0 2023-12-26T21:56:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8cf001ba00 con 0x7f8d000faed0 2023-12-26T21:56:47.473 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0010d610 con 0x7f8d000faed0 2023-12-26T21:56:47.474 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cf001a770 con 0x7f8d000faed0 2023-12-26T21:56:47.474 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.463+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0010daa0 con 0x7f8d000faed0 2023-12-26T21:56:47.474 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.464+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d00045bb0 con 0x7f8d000faed0 2023-12-26T21:56:47.474 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.465+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 11) v1 ==== 37028+0+0 (secure 0 0 0) 0x7f8cf002b070 con 0x7f8d000faed0 2023-12-26T21:56:47.474 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.465+0000 7f8cfcff9700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cec030f30 0x7f8cec0333f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:47.475 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.465+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8cf0043240 con 0x7f8d000faed0 2023-12-26T21:56:47.475 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.465+0000 7f8cff7fe700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cec030f30 0x7f8cec0333f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:47.475 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.465+0000 7f8cff7fe700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cec030f30 0x7f8cec0333f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8cf40096a0 tx=0x7f8cf4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:47.476 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.469+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8cf0023ba0 con 0x7f8d000faed0 2023-12-26T21:56:47.634 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.634+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7f8d0010e300 con 0x7f8cec030f30 2023-12-26T21:56:47.735 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.735+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f8cf000f070 con 0x7f8d000faed0 2023-12-26T21:56:47.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.749+0000 7f8cfcff9700 1 -- 172.21.15.59:0/564130968 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7f8d0010e300 con 0x7f8cec030f30 2023-12-26T21:56:47.750 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: Self-signed certificate created 2023-12-26T21:56:47.752 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cec030f30 msgr2=0x7f8cec0333f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:47.752 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cec030f30 0x7f8cec0333f0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f8cf40096a0 tx=0x7f8cf4006b40).stop 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 msgr2=0x7f8d0010cf20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d0010cf20 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8cf0011de0 tx=0x7f8cf0006ee0).stop 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 shutdown_connections 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cec030f30 0x7f8cec0333f0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 --2- 172.21.15.59:0/564130968 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8d000faed0 0x7f8d0010cf20 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 >> 172.21.15.59:0/564130968 conn(0x7f8d000f6680 msgr2=0x7f8d00104020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 shutdown_connections 2023-12-26T21:56:47.753 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:47.751+0000 7f8d06648700 1 -- 172.21.15.59:0/564130968 wait complete. 2023-12-26T21:56:47.776 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:47 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:46.323024+0000 mgr.smithi059.gwkiyc (mgr.14156) 1 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-12-26T21:56:47.776 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:47 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:46.323907+0000 mon.smithi059 (mon.0) 88 : cluster [DBG] mgrmap e11: smithi059.gwkiyc(active, since 1.0603s) 2023-12-26T21:56:47.777 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:47 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:46.330116+0000 mgr.smithi059.gwkiyc (mgr.14156) 2 : audit [DBG] from='client.14160 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-12-26T21:56:48.134 INFO:teuthology.orchestra.run.smithi059.stderr:Creating initial admin user... 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadc0044700 1 Processor -- start 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadc0044700 1 -- start start 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadc0044700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb80fb2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadc0044700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb80fb7e0 con 0x7fadb80faed0 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadbdde0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb80fb2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadbdde0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb80fb2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36568/0 (socket says 172.21.15.59:36568) 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.487+0000 7fadbdde0700 1 -- 172.21.15.59:0/223465906 learned_addr learned my addr 172.21.15.59:0/223465906 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:48.489 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadbdde0700 1 -- 172.21.15.59:0/223465906 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb80fb920 con 0x7fadb80faed0 2023-12-26T21:56:48.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadbdde0700 1 --2- 172.21.15.59:0/223465906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb80fb2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fadb400b5f0 tx=0x7fadb4006ee0).ready entity=mon.0 client_cookie=2a02ad5303784162 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:48.490 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadbcdde700 1 -- 172.21.15.59:0/223465906 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb400d600 con 0x7fadb80faed0 2023-12-26T21:56:48.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadbcdde700 1 -- 172.21.15.59:0/223465906 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fadb400dbf0 con 0x7fadb80faed0 2023-12-26T21:56:48.492 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadbcdde700 1 -- 172.21.15.59:0/223465906 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb4011bf0 con 0x7fadb80faed0 2023-12-26T21:56:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadc0044700 1 -- 172.21.15.59:0/223465906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 msgr2=0x7fadb80fb2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.488+0000 7fadc0044700 1 --2- 172.21.15.59:0/223465906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb80fb2f0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fadb400b5f0 tx=0x7fadb4006ee0).stop 2023-12-26T21:56:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 -- 172.21.15.59:0/223465906 shutdown_connections 2023-12-26T21:56:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 --2- 172.21.15.59:0/223465906 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb80fb2f0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 -- 172.21.15.59:0/223465906 >> 172.21.15.59:0/223465906 conn(0x7fadb80f6680 msgr2=0x7fadb80f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:48.493 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 -- 172.21.15.59:0/223465906 shutdown_connections 2023-12-26T21:56:48.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 -- 172.21.15.59:0/223465906 wait complete. 2023-12-26T21:56:48.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 Processor -- start 2023-12-26T21:56:48.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 -- start start 2023-12-26T21:56:48.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb8109cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:48.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.489+0000 7fadc0044700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadb810a1b0 con 0x7fadb80faed0 2023-12-26T21:56:48.494 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadbdde0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb8109cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:48.495 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadbdde0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb8109cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36574/0 (socket says 172.21.15.59:36574) 2023-12-26T21:56:48.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadbdde0700 1 -- 172.21.15.59:0/2891977131 learned_addr learned my addr 172.21.15.59:0/2891977131 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:48.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadbdde0700 1 -- 172.21.15.59:0/2891977131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadb400b200 con 0x7fadb80faed0 2023-12-26T21:56:48.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadbdde0700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb8109cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fadb4006800 tx=0x7fadb4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:48.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb400dec0 con 0x7fadb80faed0 2023-12-26T21:56:48.496 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fadb401b8a0 con 0x7fadb80faed0 2023-12-26T21:56:48.497 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadb810a3b0 con 0x7fadb80faed0 2023-12-26T21:56:48.497 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fadb401a7b0 con 0x7fadb80faed0 2023-12-26T21:56:48.497 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.490+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadb810af60 con 0x7fadb80faed0 2023-12-26T21:56:48.497 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.491+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fadb402b070 con 0x7fadb80faed0 2023-12-26T21:56:48.497 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.491+0000 7fada67fc700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fada0030ff0 0x7fada00334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:48.497 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.491+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fadb4043ae0 con 0x7fadb80faed0 2023-12-26T21:56:48.498 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.491+0000 7fadbd5df700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fada0030ff0 0x7fada00334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:48.498 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.492+0000 7fadbd5df700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fada0030ff0 0x7fada00334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fada80096a0 tx=0x7fada8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:48.498 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.492+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadb80591e0 con 0x7fadb80faed0 2023-12-26T21:56:48.499 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.496+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fadb4016080 con 0x7fadb80faed0 2023-12-26T21:56:48.674 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.673+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7fadb8059000 con 0x7fada0030ff0 2023-12-26T21:56:48.919 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.919+0000 7fada67fc700 1 -- 172.21.15.59:0/2891977131 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7fadb8059000 con 0x7fada0030ff0 2023-12-26T21:56:48.920 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$evSKOXNwfkOHhZQWEY.MwOAFniu6ZlEyAM7VKf8GCJ6JZ1VW.cO4.", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1703627808, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fada0030ff0 msgr2=0x7fada00334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fada0030ff0 0x7fada00334b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fada80096a0 tx=0x7fada8006b40).stop 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 msgr2=0x7fadb8109cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb8109cc0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7fadb4006800 tx=0x7fadb4006ee0).stop 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 shutdown_connections 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fada0030ff0 0x7fada00334b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 --2- 172.21.15.59:0/2891977131 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadb80faed0 0x7fadb8109cc0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.921+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 >> 172.21.15.59:0/2891977131 conn(0x7fadb80f6680 msgr2=0x7fadb80f8150 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.922+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 shutdown_connections 2023-12-26T21:56:48.922 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:48.922+0000 7fadc0044700 1 -- 172.21.15.59:0/2891977131 wait complete. 2023-12-26T21:56:49.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:47.635710+0000 mgr.smithi059.gwkiyc (mgr.14156) 3 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:49.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:47.737034+0000 mon.smithi059 (mon.0) 89 : cluster [DBG] mgrmap e12: smithi059.gwkiyc(active, since 2s) 2023-12-26T21:56:49.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:47.746560+0000 mon.smithi059 (mon.0) 90 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:49.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:47.750677+0000 mon.smithi059 (mon.0) 91 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:49.276 INFO:teuthology.orchestra.run.smithi059.stderr:Fetching dashboard port number... 2023-12-26T21:56:49.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.657+0000 7fdf23b38700 1 Processor -- start 2023-12-26T21:56:49.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.657+0000 7fdf23b38700 1 -- start start 2023-12-26T21:56:49.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.657+0000 7fdf23b38700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c0febe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:49.658 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.657+0000 7fdf23b38700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c0ff0d0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf218d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c0febe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:49.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf218d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c0febe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36580/0 (socket says 172.21.15.59:36580) 2023-12-26T21:56:49.661 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf218d4700 1 -- 172.21.15.59:0/3893011545 learned_addr learned my addr 172.21.15.59:0/3893011545 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf218d4700 1 -- 172.21.15.59:0/3893011545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1c0ff210 con 0x7fdf1c06ade0 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf218d4700 1 --2- 172.21.15.59:0/3893011545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c0febe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdf1000b5f0 tx=0x7fdf10006ee0).ready entity=mon.0 client_cookie=3a4927304833e49b server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf208d2700 1 -- 172.21.15.59:0/3893011545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf1000d600 con 0x7fdf1c06ade0 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf208d2700 1 -- 172.21.15.59:0/3893011545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf1000dbf0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.658+0000 7fdf208d2700 1 -- 172.21.15.59:0/3893011545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf10011bf0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 -- 172.21.15.59:0/3893011545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 msgr2=0x7fdf1c0febe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 --2- 172.21.15.59:0/3893011545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c0febe0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fdf1000b5f0 tx=0x7fdf10006ee0).stop 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 -- 172.21.15.59:0/3893011545 shutdown_connections 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 --2- 172.21.15.59:0/3893011545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c0febe0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 -- 172.21.15.59:0/3893011545 >> 172.21.15.59:0/3893011545 conn(0x7fdf1c0f6680 msgr2=0x7fdf1c0f8aa0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 -- 172.21.15.59:0/3893011545 shutdown_connections 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.659+0000 7fdf23b38700 1 -- 172.21.15.59:0/3893011545 wait complete. 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf23b38700 1 Processor -- start 2023-12-26T21:56:49.662 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf23b38700 1 -- start start 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf23b38700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c10cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf23b38700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf1c10d3f0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf218d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c10cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf218d4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c10cf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36590/0 (socket says 172.21.15.59:36590) 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf218d4700 1 -- 172.21.15.59:0/2604158660 learned_addr learned my addr 172.21.15.59:0/2604158660 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf218d4700 1 -- 172.21.15.59:0/2604158660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf1000b200 con 0x7fdf1c06ade0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf218d4700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c10cf00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fdf10011de0 tx=0x7fdf10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf1001b410 con 0x7fdf1c06ade0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdf1001ba00 con 0x7fdf1c06ade0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.660+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf1c10d5f0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.661+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf1001a770 con 0x7fdf1c06ade0 2023-12-26T21:56:49.663 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.661+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf1c10da80 con 0x7fdf1c06ade0 2023-12-26T21:56:49.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.662+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf1c045bb0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.664 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.662+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7fdf1002b070 con 0x7fdf1c06ade0 2023-12-26T21:56:49.665 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.662+0000 7fdf0e7fc700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdf04031040 0x7fdf04033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:49.665 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.662+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdf10042c00 con 0x7fdf1c06ade0 2023-12-26T21:56:49.665 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.662+0000 7fdf210d3700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdf04031040 0x7fdf04033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:49.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.662+0000 7fdf210d3700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdf04031040 0x7fdf04033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdf180096a0 tx=0x7fdf18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:49.669 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.666+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdf1001b570 con 0x7fdf1c06ade0 2023-12-26T21:56:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.825+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fdf1c0ffed0 con 0x7fdf1c06ade0 2023-12-26T21:56:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.826+0000 7fdf0e7fc700 1 -- 172.21.15.59:0/2604158660 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fdf10016020 con 0x7fdf1c06ade0 2023-12-26T21:56:49.826 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 8443 2023-12-26T21:56:49.828 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdf04031040 msgr2=0x7fdf04033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:49.828 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdf04031040 0x7fdf04033500 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdf180096a0 tx=0x7fdf18006b40).stop 2023-12-26T21:56:49.828 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 msgr2=0x7fdf1c10cf00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:49.828 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c10cf00 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fdf10011de0 tx=0x7fdf10006ee0).stop 2023-12-26T21:56:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 shutdown_connections 2023-12-26T21:56:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdf04031040 0x7fdf04033500 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 --2- 172.21.15.59:0/2604158660 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdf1c06ade0 0x7fdf1c10cf00 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 >> 172.21.15.59:0/2604158660 conn(0x7fdf1c0f6680 msgr2=0x7fdf1c0f8090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 shutdown_connections 2023-12-26T21:56:49.829 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:49.827+0000 7fdf23b38700 1 -- 172.21.15.59:0/2604158660 wait complete. 2023-12-26T21:56:50.158 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:49 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:48.675200+0000 mgr.smithi059.gwkiyc (mgr.14156) 4 : audit [DBG] from='client.14168 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:50.158 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:49 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:48.920155+0000 mon.smithi059 (mon.0) 92 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:50.158 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:49 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:49.826983+0000 mon.smithi059 (mon.0) 93 : audit [DBG] from='client.? 172.21.15.59:0/2604158660' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-12-26T21:56:50.340 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: disabled 2023-12-26T21:56:50.348 INFO:teuthology.orchestra.run.smithi059.stderr:systemctl: inactive 2023-12-26T21:56:50.349 INFO:teuthology.orchestra.run.smithi059.stderr:firewalld.service is not enabled 2023-12-26T21:56:50.349 INFO:teuthology.orchestra.run.smithi059.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-12-26T21:56:50.351 INFO:teuthology.orchestra.run.smithi059.stderr:Ceph Dashboard is now available at: 2023-12-26T21:56:50.351 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:50.352 INFO:teuthology.orchestra.run.smithi059.stderr: URL: https://smithi059.front.sepia.ceph.com:8443/ 2023-12-26T21:56:50.352 INFO:teuthology.orchestra.run.smithi059.stderr: User: admin 2023-12-26T21:56:50.352 INFO:teuthology.orchestra.run.smithi059.stderr: Password: 07ri1spltw 2023-12-26T21:56:50.352 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:50.352 INFO:teuthology.orchestra.run.smithi059.stderr:Enabling autotune for osd_memory_target 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.740+0000 7f13b42be700 1 Processor -- start 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.740+0000 7f13b42be700 1 -- start start 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.740+0000 7f13b42be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac0fc940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.740+0000 7f13b42be700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac0fce30 con 0x7f13ac0fc520 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b205a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac0fc940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b205a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac0fc940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36604/0 (socket says 172.21.15.59:36604) 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b205a700 1 -- 172.21.15.59:0/3127752012 learned_addr learned my addr 172.21.15.59:0/3127752012 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b205a700 1 -- 172.21.15.59:0/3127752012 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac0fcf70 con 0x7f13ac0fc520 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b205a700 1 --2- 172.21.15.59:0/3127752012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac0fc940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f139c00b870 tx=0x7f139c006f70).ready entity=mon.0 client_cookie=458d8c4c89b7018e server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:50.742 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b1058700 1 -- 172.21.15.59:0/3127752012 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f139c00da20 con 0x7f13ac0fc520 2023-12-26T21:56:50.743 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b1058700 1 -- 172.21.15.59:0/3127752012 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f139c010470 con 0x7f13ac0fc520 2023-12-26T21:56:50.743 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.741+0000 7f13b1058700 1 -- 172.21.15.59:0/3127752012 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f139c00f5b0 con 0x7f13ac0fc520 2023-12-26T21:56:50.746 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 -- 172.21.15.59:0/3127752012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 msgr2=0x7f13ac0fc940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:50.746 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 --2- 172.21.15.59:0/3127752012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac0fc940 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f139c00b870 tx=0x7f139c006f70).stop 2023-12-26T21:56:50.746 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 -- 172.21.15.59:0/3127752012 shutdown_connections 2023-12-26T21:56:50.746 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 --2- 172.21.15.59:0/3127752012 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac0fc940 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 -- 172.21.15.59:0/3127752012 >> 172.21.15.59:0/3127752012 conn(0x7f13ac0f5e20 msgr2=0x7f13ac0f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 -- 172.21.15.59:0/3127752012 shutdown_connections 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 -- 172.21.15.59:0/3127752012 wait complete. 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 Processor -- start 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.742+0000 7f13b42be700 1 -- start start 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b42be700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac10cf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b42be700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13ac10d480 con 0x7f13ac0fc520 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b205a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac10cf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b205a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac10cf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36606/0 (socket says 172.21.15.59:36606) 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b205a700 1 -- 172.21.15.59:0/1896559512 learned_addr learned my addr 172.21.15.59:0/1896559512 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b205a700 1 -- 172.21.15.59:0/1896559512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139c00b480 con 0x7f13ac0fc520 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b205a700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac10cf90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f139c00b840 tx=0x7f139c006f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:50.747 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f139c00f870 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f139c00fe60 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13ac10d680 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f139c019c60 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.743+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13ac10db10 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.744+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 12) v1 ==== 37108+0+0 (secure 0 0 0) 0x7f139c02a070 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.744+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13ac045bb0 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.744+0000 7f139affd700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1394030ff0 0x7f13940334b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.744+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f139c042ed0 con 0x7f13ac0fc520 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.745+0000 7f13b1859700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1394030ff0 0x7f13940334b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:50.748 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.745+0000 7f13b1859700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1394030ff0 0x7f13940334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f13a80096a0 tx=0x7f13a8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:50.749 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.749+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f139c017970 con 0x7f13ac0fc520 2023-12-26T21:56:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.907+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7f13ac0faaa0 con 0x7f13ac0fc520 2023-12-26T21:56:50.908 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.908+0000 7f139affd700 1 -- 172.21.15.59:0/1896559512 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v10)=0 v10) v1 ==== 127+0+0 (secure 0 0 0) 0x7f139c0176e0 con 0x7f13ac0fc520 2023-12-26T21:56:50.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1394030ff0 msgr2=0x7f13940334b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:50.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1394030ff0 0x7f13940334b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f13a80096a0 tx=0x7f13a8006b40).stop 2023-12-26T21:56:50.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 msgr2=0x7f13ac10cf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:50.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac10cf90 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f139c00b840 tx=0x7f139c006f70).stop 2023-12-26T21:56:50.910 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 shutdown_connections 2023-12-26T21:56:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1394030ff0 0x7f13940334b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 --2- 172.21.15.59:0/1896559512 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13ac0fc520 0x7f13ac10cf90 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 >> 172.21.15.59:0/1896559512 conn(0x7f13ac0f5e20 msgr2=0x7f13ac0ffd70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 shutdown_connections 2023-12-26T21:56:50.911 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:50.909+0000 7f13b42be700 1 -- 172.21.15.59:0/1896559512 wait complete. 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.961+0000 7fb4adeae700 1 Processor -- start 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.961+0000 7fb4adeae700 1 -- start start 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4adeae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a80fa7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4adeae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a80facd0 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a80fa7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a80fa7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36622/0 (socket says 172.21.15.59:36622) 2023-12-26T21:56:51.964 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4a77fe700 1 -- 172.21.15.59:0/696327142 learned_addr learned my addr 172.21.15.59:0/696327142 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4a77fe700 1 -- 172.21.15.59:0/696327142 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a80fae10 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4a77fe700 1 --2- 172.21.15.59:0/696327142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a80fa7e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fb49800b5f0 tx=0x7fb498006ee0).ready entity=mon.0 client_cookie=24b443b9dac42911 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.962+0000 7fb4a67fc700 1 -- 172.21.15.59:0/696327142 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49800d600 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4a67fc700 1 -- 172.21.15.59:0/696327142 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb49800dbf0 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4a67fc700 1 -- 172.21.15.59:0/696327142 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb498011bf0 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 -- 172.21.15.59:0/696327142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 msgr2=0x7fb4a80fa7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 --2- 172.21.15.59:0/696327142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a80fa7e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fb49800b5f0 tx=0x7fb498006ee0).stop 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 -- 172.21.15.59:0/696327142 shutdown_connections 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 --2- 172.21.15.59:0/696327142 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a80fa7e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 -- 172.21.15.59:0/696327142 >> 172.21.15.59:0/696327142 conn(0x7fb4a80f5e20 msgr2=0x7fb4a80f8240 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:51.965 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 -- 172.21.15.59:0/696327142 shutdown_connections 2023-12-26T21:56:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.963+0000 7fb4adeae700 1 -- 172.21.15.59:0/696327142 wait complete. 2023-12-26T21:56:51.966 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4adeae700 1 Processor -- start 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4adeae700 1 -- start start 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4adeae700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a810ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4adeae700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a810b300 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a810ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4a77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a810ae10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:36628/0 (socket says 172.21.15.59:36628) 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.964+0000 7fb4a77fe700 1 -- 172.21.15.59:0/4118898674 learned_addr learned my addr 172.21.15.59:0/4118898674 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb4a77fe700 1 -- 172.21.15.59:0/4118898674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb49800b200 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb4a77fe700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a810ae10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fb49800bdc0 tx=0x7fb498006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49800dec0 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4a810b500 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.968 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb49801b8a0 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb49801a680 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.965+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a810c110 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.966+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb49802b070 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.966+0000 7fb49ffff700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb488031090 0x7fb488033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.966+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a80591e0 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.966+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb498042c80 con 0x7fb4a80fa3c0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.966+0000 7fb4a6ffd700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb488031090 0x7fb488033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:51.969 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.967+0000 7fb4a6ffd700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb488031090 0x7fb488033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb4900096a0 tx=0x7fb490006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:51.971 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:51.971+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb49801f070 con 0x7fb4a80fa3c0 2023-12-26T21:56:52.219 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.218+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7fb4a810cf40 con 0x7fb4a80fa3c0 2023-12-26T21:56:52.220 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.220+0000 7fb49ffff700 1 -- 172.21.15.59:0/4118898674 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v31)=0 set mgr/dashboard/cluster/status v31) v1 ==== 153+0+0 (secure 0 0 0) 0x7fb498016030 con 0x7fb4a80fa3c0 2023-12-26T21:56:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb488031090 msgr2=0x7fb488033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb488031090 0x7fb488033550 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fb4900096a0 tx=0x7fb490006b40).stop 2023-12-26T21:56:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 msgr2=0x7fb4a810ae10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:52.223 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a810ae10 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fb49800bdc0 tx=0x7fb498006ee0).stop 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 shutdown_connections 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb488031090 0x7fb488033550 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 --2- 172.21.15.59:0/4118898674 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb4a80fa3c0 0x7fb4a810ae10 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 >> 172.21.15.59:0/4118898674 conn(0x7fb4a80f5e20 msgr2=0x7fb4a80f78c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 shutdown_connections 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: 2023-12-26T21:56:52.222+0000 7fb4adeae700 1 -- 172.21.15.59:0/4118898674 wait complete. 2023-12-26T21:56:52.224 INFO:teuthology.orchestra.run.smithi059.stderr:/usr/bin/ceph: set mgr/dashboard/cluster/status 2023-12-26T21:56:52.544 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:56:51.340336+0000 mon.smithi059 (mon.0) 94 : cluster [DBG] mgrmap e13: smithi059.gwkiyc(active, since 6s) 2023-12-26T21:56:52.545 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:52.221747+0000 mon.smithi059 (mon.0) 95 : audit [INF] from='client.? 172.21.15.59:0/4118898674' entity='client.admin' 2023-12-26T21:56:52.618 INFO:teuthology.orchestra.run.smithi059.stderr:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2023-12-26T21:56:52.618 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.618 INFO:teuthology.orchestra.run.smithi059.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-12-26T21:56:52.618 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.618 INFO:teuthology.orchestra.run.smithi059.stderr:Or, if you are only running a single cluster on this host: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: sudo /home/ubuntu/cephtest/cephadm shell 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr:Please consider enabling telemetry to help improve Ceph: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: ceph telemetry on 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr:For more information see: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: https://docs.ceph.com/en/pacific/mgr/telemetry/ 2023-12-26T21:56:52.619 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T21:56:52.620 INFO:teuthology.orchestra.run.smithi059.stderr:Bootstrap complete. 2023-12-26T21:56:52.652 INFO:tasks.cephadm:Fetching config... 2023-12-26T21:56:52.652 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:56:52.653 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-12-26T21:56:52.710 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-12-26T21:56:52.710 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:56:52.711 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-12-26T21:56:52.770 INFO:tasks.cephadm:Fetching mon keyring... 2023-12-26T21:56:52.771 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:56:52.771 DEBUG:teuthology.orchestra.run.smithi059:> sudo dd if=/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/mon.smithi059/keyring of=/dev/stdout 2023-12-26T21:56:52.842 INFO:tasks.cephadm:Fetching pub ssh key... 2023-12-26T21:56:52.843 DEBUG:teuthology.orchestra.run.smithi059:> set -ex 2023-12-26T21:56:52.843 DEBUG:teuthology.orchestra.run.smithi059:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-12-26T21:56:52.901 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-12-26T21:56:52.901 DEBUG:teuthology.orchestra.run.smithi059:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7EYrusFf986BdTlXSk15ugrJYzMD/3jLkCmGI93rPTqmn+VPRiuijQIMBMUYVFDJoaAdd5nsOU7WVw30E+p/BJJLhj0/7iprjrRih0eEmoA+XYFQVDZGwtUUtHxg4MGRTSy5U46eKkVxQhNpEBGzcmP/1WqJvL8wkXfQdkmgCtaW8qeF9srlmW/brdExsXjtqCXt7qtCUfJ2fPoX82Kj9jAcGmWQXgz/4JkujNgFgSG1bx/DHc0rqINxdb1ybF2hDXPyMKPB7ikmVudSwO6e25i+x2+beivlI6WVd/J4y5rLpUfdVT2YeiWNaKI/Dviq9H+MNo0B2w7WAaLgRjjnBMPXtg7yYD3xvHqqEMsnlGRpAfsADScSdNDMDcdPn7jM7ycJ4oXpKLcgqppmDnTobb5MsYVMUEAYZW0iywleSuq669080NVfOmc1fBxQf2tM0YmZIzxRtVakfTuYdJVrdWr23mDdE238JVDyMgvlvk1vz+Hj5wokJMvYkfq1Xy0= ceph-761f1ff0-a439-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-26T21:56:52.980 INFO:teuthology.orchestra.run.smithi059.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7EYrusFf986BdTlXSk15ugrJYzMD/3jLkCmGI93rPTqmn+VPRiuijQIMBMUYVFDJoaAdd5nsOU7WVw30E+p/BJJLhj0/7iprjrRih0eEmoA+XYFQVDZGwtUUtHxg4MGRTSy5U46eKkVxQhNpEBGzcmP/1WqJvL8wkXfQdkmgCtaW8qeF9srlmW/brdExsXjtqCXt7qtCUfJ2fPoX82Kj9jAcGmWQXgz/4JkujNgFgSG1bx/DHc0rqINxdb1ybF2hDXPyMKPB7ikmVudSwO6e25i+x2+beivlI6WVd/J4y5rLpUfdVT2YeiWNaKI/Dviq9H+MNo0B2w7WAaLgRjjnBMPXtg7yYD3xvHqqEMsnlGRpAfsADScSdNDMDcdPn7jM7ycJ4oXpKLcgqppmDnTobb5MsYVMUEAYZW0iywleSuq669080NVfOmc1fBxQf2tM0YmZIzxRtVakfTuYdJVrdWr23mDdE238JVDyMgvlvk1vz+Hj5wokJMvYkfq1Xy0= ceph-761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:56:52.992 DEBUG:teuthology.orchestra.run.smithi076:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7EYrusFf986BdTlXSk15ugrJYzMD/3jLkCmGI93rPTqmn+VPRiuijQIMBMUYVFDJoaAdd5nsOU7WVw30E+p/BJJLhj0/7iprjrRih0eEmoA+XYFQVDZGwtUUtHxg4MGRTSy5U46eKkVxQhNpEBGzcmP/1WqJvL8wkXfQdkmgCtaW8qeF9srlmW/brdExsXjtqCXt7qtCUfJ2fPoX82Kj9jAcGmWQXgz/4JkujNgFgSG1bx/DHc0rqINxdb1ybF2hDXPyMKPB7ikmVudSwO6e25i+x2+beivlI6WVd/J4y5rLpUfdVT2YeiWNaKI/Dviq9H+MNo0B2w7WAaLgRjjnBMPXtg7yYD3xvHqqEMsnlGRpAfsADScSdNDMDcdPn7jM7ycJ4oXpKLcgqppmDnTobb5MsYVMUEAYZW0iywleSuq669080NVfOmc1fBxQf2tM0YmZIzxRtVakfTuYdJVrdWr23mDdE238JVDyMgvlvk1vz+Hj5wokJMvYkfq1Xy0= ceph-761f1ff0-a439-11ee-95a5-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-12-26T21:56:53.042 INFO:teuthology.orchestra.run.smithi076.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQC+7EYrusFf986BdTlXSk15ugrJYzMD/3jLkCmGI93rPTqmn+VPRiuijQIMBMUYVFDJoaAdd5nsOU7WVw30E+p/BJJLhj0/7iprjrRih0eEmoA+XYFQVDZGwtUUtHxg4MGRTSy5U46eKkVxQhNpEBGzcmP/1WqJvL8wkXfQdkmgCtaW8qeF9srlmW/brdExsXjtqCXt7qtCUfJ2fPoX82Kj9jAcGmWQXgz/4JkujNgFgSG1bx/DHc0rqINxdb1ybF2hDXPyMKPB7ikmVudSwO6e25i+x2+beivlI6WVd/J4y5rLpUfdVT2YeiWNaKI/Dviq9H+MNo0B2w7WAaLgRjjnBMPXtg7yYD3xvHqqEMsnlGRpAfsADScSdNDMDcdPn7jM7ycJ4oXpKLcgqppmDnTobb5MsYVMUEAYZW0iywleSuq669080NVfOmc1fBxQf2tM0YmZIzxRtVakfTuYdJVrdWr23mDdE238JVDyMgvlvk1vz+Hj5wokJMvYkfq1Xy0= ceph-761f1ff0-a439-11ee-95a5-87774f69a715 2023-12-26T21:56:53.055 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 -- 172.21.15.59:0/1715007451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 msgr2=0x7f0da00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 --2- 172.21.15.59:0/1715007451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da00f9160 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f0d9c009fb0 tx=0x7f0d9c009b30).stop 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 -- 172.21.15.59:0/1715007451 shutdown_connections 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 --2- 172.21.15.59:0/1715007451 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da00f9160 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 -- 172.21.15.59:0/1715007451 >> 172.21.15.59:0/1715007451 conn(0x7f0da00f0be0 msgr2=0x7f0da00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 -- 172.21.15.59:0/1715007451 shutdown_connections 2023-12-26T21:56:54.432 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.431+0000 7f0da8640700 1 -- 172.21.15.59:0/1715007451 wait complete. 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.432+0000 7f0da8640700 1 Processor -- start 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.432+0000 7f0da8640700 1 -- start start 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da8640700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da0106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da8640700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0da0106c40 con 0x7f0da0060360 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da63dc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da0106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da63dc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da0106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44106/0 (socket says 172.21.15.59:44106) 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da63dc700 1 -- 172.21.15.59:0/99011299 learned_addr learned my addr 172.21.15.59:0/99011299 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da63dc700 1 -- 172.21.15.59:0/99011299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9c005040 con 0x7f0da0060360 2023-12-26T21:56:54.434 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da63dc700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da0106750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0d9c000c00 tx=0x7f0d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:54.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c0095f0 con 0x7f0da0060360 2023-12-26T21:56:54.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0da0106e40 con 0x7f0da0060360 2023-12-26T21:56:54.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0d9c007e50 con 0x7f0da0060360 2023-12-26T21:56:54.435 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.433+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d9c014e70 con 0x7f0da0060360 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.434+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0d9c01d4e0 con 0x7f0da0060360 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.434+0000 7f0d96ffd700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0d8c031090 0x7f0d8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.434+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0da01072d0 con 0x7f0da0060360 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.434+0000 7f0da5bdb700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0d8c031090 0x7f0d8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.435+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0d9c014420 con 0x7f0da0060360 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.435+0000 7f0da5bdb700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0d8c031090 0x7f0d8c033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0d900096a0 tx=0x7f0d90006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:54.436 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.435+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0da00591e0 con 0x7f0da0060360 2023-12-26T21:56:54.444 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.443+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0d9c00e890 con 0x7f0da0060360 2023-12-26T21:56:54.619 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.619+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0da0108060 con 0x7f0da0060360 2023-12-26T21:56:54.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.622+0000 7f0d96ffd700 1 -- 172.21.15.59:0/99011299 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0d9c00e2a0 con 0x7f0da0060360 2023-12-26T21:56:54.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0d8c031090 msgr2=0x7f0d8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:54.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0d8c031090 0x7f0d8c033550 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f0d900096a0 tx=0x7f0d90006b40).stop 2023-12-26T21:56:54.625 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 msgr2=0x7f0da0106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da0106750 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0d9c000c00 tx=0x7f0d9c00b040).stop 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 shutdown_connections 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0d8c031090 0x7f0d8c033550 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 --2- 172.21.15.59:0/99011299 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0da0060360 0x7f0da0106750 secure :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0x7f0d9c000c00 tx=0x7f0d9c00b040).stop 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 >> 172.21.15.59:0/99011299 conn(0x7f0da00f0be0 msgr2=0x7f0da00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 shutdown_connections 2023-12-26T21:56:54.626 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:54.624+0000 7f0da8640700 1 -- 172.21.15.59:0/99011299 wait complete. 2023-12-26T21:56:55.014 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2023-12-26T21:56:55.014 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph orch client-keyring set client.admin '*' --mode 0755 2023-12-26T21:56:55.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:54.511882+0000 mon.smithi059 (mon.0) 96 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:55.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:54.623252+0000 mon.smithi059 (mon.0) 97 : audit [INF] from='client.? 172.21.15.59:0/99011299' entity='client.admin' 2023-12-26T21:56:55.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:54.911901+0000 mon.smithi059 (mon.0) 98 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config rm", "who": "osd/host:smithi059", "name": "osd_memory_target"}]: dispatch 2023-12-26T21:56:55.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:54.915761+0000 mon.smithi059 (mon.0) 99 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:55.792 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:54.923035+0000 mgr.smithi059.gwkiyc (mgr.14156) 5 : cephadm [INF] Deploying daemon alertmanager.smithi059 on smithi059 2023-12-26T21:56:56.378 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 -- 172.21.15.59:0/4097515460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 msgr2=0x7fa5940f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 --2- 172.21.15.59:0/4097515460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa5940f9160 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fa588009fb0 tx=0x7fa588009b30).stop 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 -- 172.21.15.59:0/4097515460 shutdown_connections 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 --2- 172.21.15.59:0/4097515460 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa5940f9160 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 -- 172.21.15.59:0/4097515460 >> 172.21.15.59:0/4097515460 conn(0x7fa5940f0be0 msgr2=0x7fa5940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 -- 172.21.15.59:0/4097515460 shutdown_connections 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 -- 172.21.15.59:0/4097515460 wait complete. 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.376+0000 7fa5997fd700 1 Processor -- start 2023-12-26T21:56:56.379 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa5997fd700 1 -- start start 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa5997fd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa594106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa5997fd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa594106c40 con 0x7fa594060360 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa592ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa594106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa592ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa594106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44128/0 (socket says 172.21.15.59:44128) 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa592ffd700 1 -- 172.21.15.59:0/1710205653 learned_addr learned my addr 172.21.15.59:0/1710205653 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa592ffd700 1 -- 172.21.15.59:0/1710205653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa588005040 con 0x7fa594060360 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa592ffd700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa594106750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa588009f80 tx=0x7fa58800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa588009320 con 0x7fa594060360 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa594106e40 con 0x7fa594060360 2023-12-26T21:56:56.380 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa588013680 con 0x7fa594060360 2023-12-26T21:56:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.377+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa58801eda0 con 0x7fa594060360 2023-12-26T21:56:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.378+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5941072d0 con 0x7fa594060360 2023-12-26T21:56:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.378+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa588014420 con 0x7fa594060360 2023-12-26T21:56:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.378+0000 7fa57b7fe700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa57c031090 0x7fa57c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.378+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa588047610 con 0x7fa594060360 2023-12-26T21:56:56.381 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.378+0000 7fa5927fc700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa57c031090 0x7fa57c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:56.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.379+0000 7fa5927fc700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa57c031090 0x7fa57c033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa5840096a0 tx=0x7fa584006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:56.382 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.379+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa5940591e0 con 0x7fa594060360 2023-12-26T21:56:56.383 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.383+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa588048060 con 0x7fa594060360 2023-12-26T21:56:56.548 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.548+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5940f1d70 con 0x7fa57c031090 2023-12-26T21:56:56.551 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.551+0000 7fa57b7fe700 1 -- 172.21.15.59:0/1710205653 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fa5940f1d70 con 0x7fa57c031090 2023-12-26T21:56:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa57c031090 msgr2=0x7fa57c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:56.553 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa57c031090 0x7fa57c033550 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa5840096a0 tx=0x7fa584006b40).stop 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 msgr2=0x7fa594106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa594106750 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa588009f80 tx=0x7fa58800b040).stop 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 shutdown_connections 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa57c031090 0x7fa57c033550 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 --2- 172.21.15.59:0/1710205653 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa594060360 0x7fa594106750 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 >> 172.21.15.59:0/1710205653 conn(0x7fa5940f0be0 msgr2=0x7fa5940f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:56.554 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 shutdown_connections 2023-12-26T21:56:56.555 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:56.552+0000 7fa5997fd700 1 -- 172.21.15.59:0/1710205653 wait complete. 2023-12-26T21:56:56.952 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi076 2023-12-26T21:56:56.952 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:56:56.952 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/etc/ceph/ceph.conf 2023-12-26T21:56:56.973 DEBUG:teuthology.orchestra.run.smithi076:> set -ex 2023-12-26T21:56:56.974 DEBUG:teuthology.orchestra.run.smithi076:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-12-26T21:56:57.035 INFO:tasks.cephadm:Adding host smithi076 to orchestrator... 2023-12-26T21:56:57.036 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph orch host add smithi076 2023-12-26T21:56:57.673 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:57 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:56.549557+0000 mgr.smithi059.gwkiyc (mgr.14156) 6 : audit [DBG] from='client.14178 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:57.674 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:57 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:56.551964+0000 mon.smithi059 (mon.0) 100 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.152+0000 7fa0b83c4700 1 -- 172.21.15.59:0/171514454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 msgr2=0x7fa0b00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.152+0000 7fa0b83c4700 1 --2- 172.21.15.59:0/171514454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b00f69b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fa0ac004760 tx=0x7fa0ac009ad0).stop 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 -- 172.21.15.59:0/171514454 shutdown_connections 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 --2- 172.21.15.59:0/171514454 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b00f69b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 -- 172.21.15.59:0/171514454 >> 172.21.15.59:0/171514454 conn(0x7fa0b00f1cd0 msgr2=0x7fa0b00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 -- 172.21.15.59:0/171514454 shutdown_connections 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 -- 172.21.15.59:0/171514454 wait complete. 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 Processor -- start 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 -- start start 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:58.155 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b83c4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa0b0106cb0 con 0x7fa0b00f6590 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b6160700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.153+0000 7fa0b6160700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44156/0 (socket says 172.21.15.59:44156) 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0b6160700 1 -- 172.21.15.59:0/1068906969 learned_addr learned my addr 172.21.15.59:0/1068906969 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0b6160700 1 -- 172.21.15.59:0/1068906969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa0ac009d70 con 0x7fa0b00f6590 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0b6160700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b01067c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa0ac004170 tx=0x7fa0ac009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac00e040 con 0x7fa0b00f6590 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa0ac008ef0 con 0x7fa0b00f6590 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa0b0106eb0 con 0x7fa0b00f6590 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa0ac014c90 con 0x7fa0b00f6590 2023-12-26T21:56:58.156 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.154+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa0b0107340 con 0x7fa0b00f6590 2023-12-26T21:56:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.155+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa0b0045bb0 con 0x7fa0b00f6590 2023-12-26T21:56:58.157 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.155+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa0ac014420 con 0x7fa0b00f6590 2023-12-26T21:56:58.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.155+0000 7fa0a6ffd700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa09c0310e0 0x7fa09c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:56:58.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.155+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa0ac0421f0 con 0x7fa0b00f6590 2023-12-26T21:56:58.158 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.155+0000 7fa0b595f700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa09c0310e0 0x7fa09c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:56:58.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.156+0000 7fa0b595f700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa09c0310e0 0x7fa09c0335a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa0a00096a0 tx=0x7fa0a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:56:58.159 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.159+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa0ac026300 con 0x7fa0b00f6590 2023-12-26T21:56:58.325 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.324+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}) v1 -- 0x7fa0b00f2ed0 con 0x7fa09c0310e0 2023-12-26T21:56:58.975 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.974+0000 7fa0a6ffd700 1 -- 172.21.15.59:0/1068906969 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+48 (secure 0 0 0) 0x7fa0b00f2ed0 con 0x7fa09c0310e0 2023-12-26T21:56:59.303 INFO:teuthology.orchestra.run.smithi059.stdout:Added host 'smithi076' with addr '172.21.15.76' 2023-12-26T21:56:59.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.976+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa09c0310e0 msgr2=0x7fa09c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:59.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.976+0000 7fa0b83c4700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa09c0310e0 0x7fa09c0335a0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fa0a00096a0 tx=0x7fa0a0006b40).stop 2023-12-26T21:56:59.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.976+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 msgr2=0x7fa0b01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:56:59.304 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.976+0000 7fa0b83c4700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b01067c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fa0ac004170 tx=0x7fa0ac009ad0).stop 2023-12-26T21:56:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.977+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 shutdown_connections 2023-12-26T21:56:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.977+0000 7fa0b83c4700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa09c0310e0 0x7fa09c0335a0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.977+0000 7fa0b83c4700 1 --2- 172.21.15.59:0/1068906969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa0b00f6590 0x7fa0b01067c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:56:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.977+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 >> 172.21.15.59:0/1068906969 conn(0x7fa0b00f1cd0 msgr2=0x7fa0b00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:56:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.977+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 shutdown_connections 2023-12-26T21:56:59.305 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:56:58.977+0000 7fa0b83c4700 1 -- 172.21.15.59:0/1068906969 wait complete. 2023-12-26T21:56:59.305 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:58.326517+0000 mgr.smithi059.gwkiyc (mgr.14156) 7 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi076", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:56:59.305 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:56:58.975392+0000 mon.smithi059 (mon.0) 101 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:56:59.393 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph orch host ls --format=json 2023-12-26T21:57:00.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:56:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:56:58.975714+0000 mgr.smithi059.gwkiyc (mgr.14156) 8 : cephadm [INF] Added host smithi076 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.568+0000 7ff0b844a700 1 -- 172.21.15.59:0/4036893998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 msgr2=0x7ff0b00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.568+0000 7ff0b844a700 1 --2- 172.21.15.59:0/4036893998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b00f58a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7ff0ac009fb0 tx=0x7ff0ac009b30).stop 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 -- 172.21.15.59:0/4036893998 shutdown_connections 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 --2- 172.21.15.59:0/4036893998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b00f58a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 -- 172.21.15.59:0/4036893998 >> 172.21.15.59:0/4036893998 conn(0x7ff0b00f0be0 msgr2=0x7ff0b00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 -- 172.21.15.59:0/4036893998 shutdown_connections 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 -- 172.21.15.59:0/4036893998 wait complete. 2023-12-26T21:57:00.571 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 Processor -- start 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 -- start start 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b01094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b844a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0b0060320 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b61e6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b01094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b61e6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b01094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:44180/0 (socket says 172.21.15.59:44180) 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.569+0000 7ff0b61e6700 1 -- 172.21.15.59:0/2252105658 learned_addr learned my addr 172.21.15.59:0/2252105658 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0b61e6700 1 -- 172.21.15.59:0/2252105658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac005040 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0b61e6700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b01094f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff0ac0089f0 tx=0x7ff0ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac009320 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0b0060520 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff0ac013ec0 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0ac014d30 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.570+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0b0109ba0 con 0x7ff0b00f5480 2023-12-26T21:57:00.572 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.571+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff0ac0263a0 con 0x7ff0b00f5480 2023-12-26T21:57:00.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.571+0000 7ff0a2ffd700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff09c0310e0 0x7ff09c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:00.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.571+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff0ac0423a0 con 0x7ff0b00f5480 2023-12-26T21:57:00.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.571+0000 7ff0b59e5700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff09c0310e0 0x7ff09c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:00.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.571+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0b00591e0 con 0x7ff0b00f5480 2023-12-26T21:57:00.573 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.571+0000 7ff0b59e5700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff09c0310e0 0x7ff09c0335a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff0a40096a0 tx=0x7ff0a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:00.576 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.576+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff0ac012920 con 0x7ff0b00f5480 2023-12-26T21:57:00.741 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.741+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7ff0b00f1de0 con 0x7ff09c0310e0 2023-12-26T21:57:00.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.742+0000 7ff0a2ffd700 1 -- 172.21.15.59:0/2252105658 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+159 (secure 0 0 0) 0x7ff0b00f1de0 con 0x7ff09c0310e0 2023-12-26T21:57:00.742 INFO:teuthology.orchestra.run.smithi059.stdout: 2023-12-26T21:57:00.743 INFO:teuthology.orchestra.run.smithi059.stdout:[{"addr": "172.21.15.59", "hostname": "smithi059", "labels": [], "status": ""}, {"addr": "172.21.15.76", "hostname": "smithi076", "labels": [], "status": ""}] 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff09c0310e0 msgr2=0x7ff09c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff09c0310e0 0x7ff09c0335a0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7ff0a40096a0 tx=0x7ff0a4006b40).stop 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 msgr2=0x7ff0b01094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b01094f0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7ff0ac0089f0 tx=0x7ff0ac00b040).stop 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 shutdown_connections 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff09c0310e0 0x7ff09c0335a0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 --2- 172.21.15.59:0/2252105658 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0b00f5480 0x7ff0b01094f0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 >> 172.21.15.59:0/2252105658 conn(0x7ff0b00f0be0 msgr2=0x7ff0b00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 shutdown_connections 2023-12-26T21:57:00.745 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:00.744+0000 7ff0b844a700 1 -- 172.21.15.59:0/2252105658 wait complete. 2023-12-26T21:57:01.026 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:00.743080+0000 mgr.smithi059.gwkiyc (mgr.14156) 9 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-12-26T21:57:01.145 INFO:tasks.cephadm:Setting crush tunables to default 2023-12-26T21:57:01.147 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph osd crush tunables default 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 -- 172.21.15.59:0/1251725161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 msgr2=0x7f05640f6e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 --2- 172.21.15.59:0/1251725161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f05640f6e80 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f0554009fb0 tx=0x7f0554009b30).stop 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 -- 172.21.15.59:0/1251725161 shutdown_connections 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 --2- 172.21.15.59:0/1251725161 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f05640f6e80 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 -- 172.21.15.59:0/1251725161 >> 172.21.15.59:0/1251725161 conn(0x7f05640f0390 msgr2=0x7f05640f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 -- 172.21.15.59:0/1251725161 shutdown_connections 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.282+0000 7f056b47a700 1 -- 172.21.15.59:0/1251725161 wait complete. 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f056b47a700 1 Processor -- start 2023-12-26T21:57:02.284 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f056b47a700 1 -- start start 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f056b47a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f0564106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f056b47a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0564106c80 con 0x7f05640f6a60 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0569216700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f0564106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0569216700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f0564106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.59:52992/0 (socket says 172.21.15.59:52992) 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0569216700 1 -- 172.21.15.59:0/3911490110 learned_addr learned my addr 172.21.15.59:0/3911490110 (peer_addr_for_me v2:172.21.15.59:0/0) 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0569216700 1 -- 172.21.15.59:0/3911490110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0554005040 con 0x7f05640f6a60 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0569216700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f0564106790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f05540089f0 tx=0x7f055400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0554009320 con 0x7f05640f6a60 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0554013ec0 con 0x7f05640f6a60 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0564106e80 con 0x7f05640f6a60 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.283+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0554014d30 con 0x7f05640f6a60 2023-12-26T21:57:02.285 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.284+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0564107310 con 0x7f05640f6a60 2023-12-26T21:57:02.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.284+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f05540263a0 con 0x7f05640f6a60 2023-12-26T21:57:02.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.285+0000 7f0559ffb700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f05500310e0 0x7f05500335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:02.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.285+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0554042380 con 0x7f05640f6a60 2023-12-26T21:57:02.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.285+0000 7f0568a15700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f05500310e0 0x7f05500335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:02.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.285+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05640591e0 con 0x7f05640f6a60 2023-12-26T21:57:02.286 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.285+0000 7f0568a15700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f05500310e0 0x7f05500335a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f05600096a0 tx=0x7f0560006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:02.289 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.289+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0554026680 con 0x7f05640f6a60 2023-12-26T21:57:02.457 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.456+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f0564045bb0 con 0x7f05640f6a60 2023-12-26T21:57:02.740 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.740+0000 7f0559ffb700 1 -- 172.21.15.59:0/3911490110 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f055402b030 con 0x7f05640f6a60 2023-12-26T21:57:02.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f05500310e0 msgr2=0x7f05500335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:02.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f05500310e0 0x7f05500335a0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f05600096a0 tx=0x7f0560006b40).stop 2023-12-26T21:57:02.742 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 msgr2=0x7f0564106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f0564106790 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f05540089f0 tx=0x7f055400b040).stop 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 shutdown_connections 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f05500310e0 0x7f05500335a0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 --2- 172.21.15.59:0/3911490110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f05640f6a60 0x7f0564106790 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 >> 172.21.15.59:0/3911490110 conn(0x7f05640f0390 msgr2=0x7f05640f4ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 shutdown_connections 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:2023-12-26T21:57:02.742+0000 7f056b47a700 1 -- 172.21.15.59:0/3911490110 wait complete. 2023-12-26T21:57:02.743 INFO:teuthology.orchestra.run.smithi059.stderr:adjusted tunables profile to default 2023-12-26T21:57:03.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:02.458213+0000 mon.smithi059 (mon.0) 102 : audit [INF] from='client.? 172.21.15.59:0/3911490110' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-12-26T21:57:03.116 INFO:tasks.cephadm:Adding mon.smithi059 on smithi059 2023-12-26T21:57:03.116 INFO:tasks.cephadm:Adding mon.smithi076 on smithi076 2023-12-26T21:57:03.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph orch apply mon '2;smithi059:172.21.15.59=smithi059;smithi076:172.21.15.76=smithi076' 2023-12-26T21:57:04.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:03 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:02.741148+0000 mon.smithi059 (mon.0) 103 : audit [INF] from='client.? 172.21.15.59:0/3911490110' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-12-26T21:57:04.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:03 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:02.741208+0000 mon.smithi059 (mon.0) 104 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-12-26T21:57:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 -- 172.21.15.76:0/4123347505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 msgr2=0x7f28940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 --2- 172.21.15.76:0/4123347505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28940f58a0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f2884009fb0 tx=0x7f2884009b30).stop 2023-12-26T21:57:04.261 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 -- 172.21.15.76:0/4123347505 shutdown_connections 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 --2- 172.21.15.76:0/4123347505 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28940f58a0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 -- 172.21.15.76:0/4123347505 >> 172.21.15.76:0/4123347505 conn(0x7f28940f0be0 msgr2=0x7f28940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 -- 172.21.15.76:0/4123347505 shutdown_connections 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.258+0000 7f2898be2700 1 -- 172.21.15.76:0/4123347505 wait complete. 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f2898be2700 1 Processor -- start 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f2898be2700 1 -- start start 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f2898be2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28941067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f2898be2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2894106cd0 con 0x7f28940f5480 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f289259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28941067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f289259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28941067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40728/0 (socket says 172.21.15.76:40728) 2023-12-26T21:57:04.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f289259c700 1 -- 172.21.15.76:0/3802754205 learned_addr learned my addr 172.21.15.76:0/3802754205 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f289259c700 1 -- 172.21.15.76:0/3802754205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2884005040 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.259+0000 7f289259c700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28941067e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f2884000c00 tx=0x7f288400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28840095f0 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2894106ed0 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2884007e50 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2884014e70 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2894107360 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2884025070 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.260+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2894045bb0 con 0x7f28940f5480 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.261+0000 7f2882ffd700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f287c031090 0x7f287c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:04.263 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.261+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2884041d40 con 0x7f28940f5480 2023-12-26T21:57:04.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.261+0000 7f2891d9b700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f287c031090 0x7f287c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:04.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.261+0000 7f2891d9b700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f287c031090 0x7f287c033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f28880096a0 tx=0x7f2888006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:04.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.265+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f288400f300 con 0x7f28940f5480 2023-12-26T21:57:04.428 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.428+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 --> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi076:172.21.15.76=smithi076", "target": ["mon-mgr", ""]}) v1 -- 0x7f28940f2af0 con 0x7f287c031090 2023-12-26T21:57:04.434 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.433+0000 7f2882ffd700 1 -- 172.21.15.76:0/3802754205 <== mgr.14156 v2:172.21.15.59:6800/1607822224 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f28940f2af0 con 0x7f287c031090 2023-12-26T21:57:04.434 INFO:teuthology.orchestra.run.smithi076.stdout:Scheduled mon update... 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f287c031090 msgr2=0x7f287c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f287c031090 0x7f287c033550 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f28880096a0 tx=0x7f2888006b40).stop 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 msgr2=0x7f28941067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28941067e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f2884000c00 tx=0x7f288400b040).stop 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 shutdown_connections 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f287c031090 0x7f287c033550 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 --2- 172.21.15.76:0/3802754205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28940f5480 0x7f28941067e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 >> 172.21.15.76:0/3802754205 conn(0x7f28940f0be0 msgr2=0x7f289405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 shutdown_connections 2023-12-26T21:57:04.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:04.435+0000 7f2898be2700 1 -- 172.21.15.76:0/3802754205 wait complete. 2023-12-26T21:57:04.808 DEBUG:teuthology.orchestra.run.smithi076:mon.smithi076> sudo journalctl -f -n 0 -u ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076.service 2023-12-26T21:57:04.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:04.810 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:04.839 INFO:journalctl@ceph.mon.smithi076.smithi076.stdout:-- Logs begin at Tue 2023-12-26 21:42:46 UTC. -- 2023-12-26T21:57:05.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:05 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:04.430451+0000 mgr.smithi059.gwkiyc (mgr.14156) 10 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi059:172.21.15.59=smithi059;smithi076:172.21.15.76=smithi076", "target": ["mon-mgr", ""]}]: dispatch 2023-12-26T21:57:05.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:05 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T21:57:04.431962+0000 mgr.smithi059.gwkiyc (mgr.14156) 11 : cephadm [INF] Saving service mon spec with placement smithi059:172.21.15.59=smithi059;smithi076:172.21.15.76=smithi076;count:2 2023-12-26T21:57:05.795 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:05 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:04.434714+0000 mon.smithi059 (mon.0) 105 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 -- 172.21.15.76:0/620246959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 msgr2=0x7f76040f6cf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 --2- 172.21.15.76:0/620246959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f76040f6cf0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f75f8009fb0 tx=0x7f75f8009b30).stop 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 -- 172.21.15.76:0/620246959 shutdown_connections 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 --2- 172.21.15.76:0/620246959 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f76040f6cf0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 -- 172.21.15.76:0/620246959 >> 172.21.15.76:0/620246959 conn(0x7f76040f01d0 msgr2=0x7f76040f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 -- 172.21.15.76:0/620246959 shutdown_connections 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 -- 172.21.15.76:0/620246959 wait complete. 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 Processor -- start 2023-12-26T21:57:06.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.023+0000 7f7609bd1700 1 -- start start 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.024+0000 7f7609bd1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f7604109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.024+0000 7f7609bd1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76040f5260 con 0x7f76040f68d0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.024+0000 7f7608bcf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f7604109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.024+0000 7f7608bcf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f7604109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40060/0 (socket says 172.21.15.76:40060) 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.024+0000 7f7608bcf700 1 -- 172.21.15.76:0/3210996287 learned_addr learned my addr 172.21.15.76:0/3210996287 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.024+0000 7f7608bcf700 1 -- 172.21.15.76:0/3210996287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75f8005040 con 0x7f76040f68d0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.025+0000 7f7608bcf700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f7604109330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f75f80089f0 tx=0x7f75f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.025+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f8009320 con 0x7f76040f68d0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.025+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76040f5460 con 0x7f76040f68d0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.025+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f75f8013ec0 con 0x7f76040f68d0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.025+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75f8014d30 con 0x7f76040f68d0 2023-12-26T21:57:06.027 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.025+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76041099e0 con 0x7f76040f68d0 2023-12-26T21:57:06.028 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.026+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f75f80263a0 con 0x7f76040f68d0 2023-12-26T21:57:06.028 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.026+0000 7f76017fa700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f75f4031090 0x7f75f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:06.028 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.026+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f75f80424e0 con 0x7f76040f68d0 2023-12-26T21:57:06.028 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.026+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76040591e0 con 0x7f76040f68d0 2023-12-26T21:57:06.029 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.026+0000 7f7603fff700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f75f4031090 0x7f75f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:06.029 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.026+0000 7f7603fff700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f75f4031090 0x7f75f4033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f75f00096a0 tx=0x7f75f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:06.031 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.031+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f75f80266d0 con 0x7f76040f68d0 2023-12-26T21:57:06.238 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.238+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7604045bb0 con 0x7f76040f68d0 2023-12-26T21:57:06.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.238+0000 7f76017fa700 1 -- 172.21.15.76:0/3210996287 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f75f802b020 con 0x7f76040f68d0 2023-12-26T21:57:06.239 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:06.239 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:06.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.239+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f75f4031090 msgr2=0x7f75f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:06.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.239+0000 7f7609bd1700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f75f4031090 0x7f75f4033550 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f75f00096a0 tx=0x7f75f0006b40).stop 2023-12-26T21:57:06.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.239+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 msgr2=0x7f7604109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:06.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.239+0000 7f7609bd1700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f7604109330 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f75f80089f0 tx=0x7f75f800b040).stop 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.240+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 shutdown_connections 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.240+0000 7f7609bd1700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f75f4031090 0x7f75f4033550 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.240+0000 7f7609bd1700 1 --2- 172.21.15.76:0/3210996287 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76040f68d0 0x7f7604109330 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.240+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 >> 172.21.15.76:0/3210996287 conn(0x7f76040f01d0 msgr2=0x7f76040f0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.240+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 shutdown_connections 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:06.240+0000 7f7609bd1700 1 -- 172.21.15.76:0/3210996287 wait complete. 2023-12-26T21:57:06.241 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:06.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:05.324552+0000 mgr.smithi059.gwkiyc (mgr.14156) 12 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:06.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:06.239776+0000 mon.smithi059 (mon.0) 106 : audit [DBG] from='client.? 172.21.15.76:0/3210996287' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:07.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:07.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 -- 172.21.15.76:0/556438485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 msgr2=0x7fee140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 --2- 172.21.15.76:0/556438485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee140f58a0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fee04009fb0 tx=0x7fee04009b30).stop 2023-12-26T21:57:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 -- 172.21.15.76:0/556438485 shutdown_connections 2023-12-26T21:57:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 --2- 172.21.15.76:0/556438485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee140f58a0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 -- 172.21.15.76:0/556438485 >> 172.21.15.76:0/556438485 conn(0x7fee140f0be0 msgr2=0x7fee140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:08.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 -- 172.21.15.76:0/556438485 shutdown_connections 2023-12-26T21:57:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 -- 172.21.15.76:0/556438485 wait complete. 2023-12-26T21:57:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.851+0000 7fee19149700 1 Processor -- start 2023-12-26T21:57:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee19149700 1 -- start start 2023-12-26T21:57:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee19149700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee141067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:08.853 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee19149700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee14106cd0 con 0x7fee140f5480 2023-12-26T21:57:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee12d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee141067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee12d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee141067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40078/0 (socket says 172.21.15.76:40078) 2023-12-26T21:57:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee12d9d700 1 -- 172.21.15.76:0/4261758282 learned_addr learned my addr 172.21.15.76:0/4261758282 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:08.854 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee12d9d700 1 -- 172.21.15.76:0/4261758282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee04005040 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.852+0000 7fee12d9d700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee141067e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fee040089f0 tx=0x7fee0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee04009320 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee14106ed0 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fee04013ec0 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee04014d30 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee14107360 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fee040263a0 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.853+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee14045bb0 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.854+0000 7fedfb7fe700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fedfc031090 0x7fedfc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.854+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fee04042460 con 0x7fee140f5480 2023-12-26T21:57:08.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.854+0000 7fee1259c700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fedfc031090 0x7fedfc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:08.856 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.854+0000 7fee1259c700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fedfc031090 0x7fedfc033550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fee080096a0 tx=0x7fee08006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:08.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:08.859+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fee0401db70 con 0x7fee140f5480 2023-12-26T21:57:09.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:07.324870+0000 mgr.smithi059.gwkiyc (mgr.14156) 13 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:09.069 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.068+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee14108210 con 0x7fee140f5480 2023-12-26T21:57:09.070 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.069+0000 7fedfb7fe700 1 -- 172.21.15.76:0/4261758282 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee0401db70 con 0x7fee140f5480 2023-12-26T21:57:09.070 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:09.070 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:09.071 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.070+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fedfc031090 msgr2=0x7fedfc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:09.071 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.070+0000 7fee19149700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fedfc031090 0x7fedfc033550 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7fee080096a0 tx=0x7fee08006b40).stop 2023-12-26T21:57:09.071 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.070+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 msgr2=0x7fee141067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:09.071 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.070+0000 7fee19149700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee141067e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fee040089f0 tx=0x7fee0400b040).stop 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.071+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 shutdown_connections 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.071+0000 7fee19149700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fedfc031090 0x7fedfc033550 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.071+0000 7fee19149700 1 --2- 172.21.15.76:0/4261758282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee140f5480 0x7fee141067e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.071+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 >> 172.21.15.76:0/4261758282 conn(0x7fee140f0be0 msgr2=0x7fee1405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.071+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 shutdown_connections 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:09.071+0000 7fee19149700 1 -- 172.21.15.76:0/4261758282 wait complete. 2023-12-26T21:57:09.072 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:10.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:09 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:09.070722+0000 mon.smithi059 (mon.0) 107 : audit [DBG] from='client.? 172.21.15.76:0/4261758282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:10.462 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:10.462 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:11.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:09.325240+0000 mgr.smithi059.gwkiyc (mgr.14156) 14 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:11.656 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 -- 172.21.15.76:0/1619867899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 msgr2=0x7fe2900f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 --2- 172.21.15.76:0/1619867899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2900f4c60 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7fe284009fb0 tx=0x7fe284009b30).stop 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 -- 172.21.15.76:0/1619867899 shutdown_connections 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 --2- 172.21.15.76:0/1619867899 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2900f4c60 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 -- 172.21.15.76:0/1619867899 >> 172.21.15.76:0/1619867899 conn(0x7fe2900f0360 msgr2=0x7fe2900f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 -- 172.21.15.76:0/1619867899 shutdown_connections 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.655+0000 7fe29702f700 1 -- 172.21.15.76:0/1619867899 wait complete. 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.656+0000 7fe29702f700 1 Processor -- start 2023-12-26T21:57:11.657 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.656+0000 7fe29702f700 1 -- start start 2023-12-26T21:57:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.656+0000 7fe29702f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2901067b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe29702f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe290106ca0 con 0x7fe2900f4840 2023-12-26T21:57:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe294dcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2901067b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:11.659 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe294dcb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2901067b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40090/0 (socket says 172.21.15.76:40090) 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe294dcb700 1 -- 172.21.15.76:0/389195600 learned_addr learned my addr 172.21.15.76:0/389195600 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe294dcb700 1 -- 172.21.15.76:0/389195600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe284005040 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe294dcb700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2901067b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe2840089f0 tx=0x7fe28400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe284009320 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe290106ea0 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe284013ec0 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe284014d30 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.657+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe290107330 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.658+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe284014420 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.658+0000 7fe28d7fa700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe280031090 0x7fe280033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.658+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2840424a0 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.658+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe290045bb0 con 0x7fe2900f4840 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.659+0000 7fe28ffff700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe280031090 0x7fe280033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:11.660 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.659+0000 7fe28ffff700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe280031090 0x7fe280033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fe27c0096a0 tx=0x7fe27c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:11.664 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.663+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe284014750 con 0x7fe2900f4840 2023-12-26T21:57:11.870 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.869+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe290108070 con 0x7fe2900f4840 2023-12-26T21:57:11.870 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.870+0000 7fe28d7fa700 1 -- 172.21.15.76:0/389195600 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe28401d7f0 con 0x7fe2900f4840 2023-12-26T21:57:11.870 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:11.871 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:11.872 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe280031090 msgr2=0x7fe280033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:11.872 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe280031090 0x7fe280033550 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fe27c0096a0 tx=0x7fe27c006b40).stop 2023-12-26T21:57:11.872 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 msgr2=0x7fe2901067b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:11.872 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2901067b0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fe2840089f0 tx=0x7fe28400b040).stop 2023-12-26T21:57:11.872 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 shutdown_connections 2023-12-26T21:57:11.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe280031090 0x7fe280033550 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:11.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 --2- 172.21.15.76:0/389195600 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2900f4840 0x7fe2901067b0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:11.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 >> 172.21.15.76:0/389195600 conn(0x7fe2900f0360 msgr2=0x7fe2900f0e40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:11.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.871+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 shutdown_connections 2023-12-26T21:57:11.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:11.872+0000 7fe29702f700 1 -- 172.21.15.76:0/389195600 wait complete. 2023-12-26T21:57:11.873 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:13.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:11.325614+0000 mgr.smithi059.gwkiyc (mgr.14156) 15 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:13.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:11.871408+0000 mon.smithi059 (mon.0) 108 : audit [DBG] from='client.? 172.21.15.76:0/389195600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:13.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:13.258 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:14.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 -- 172.21.15.76:0/1366465490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 msgr2=0x7f22580f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:14.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 --2- 172.21.15.76:0/1366465490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22580f58a0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f224c009fb0 tx=0x7f224c009b30).stop 2023-12-26T21:57:14.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 -- 172.21.15.76:0/1366465490 shutdown_connections 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 --2- 172.21.15.76:0/1366465490 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22580f58a0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 -- 172.21.15.76:0/1366465490 >> 172.21.15.76:0/1366465490 conn(0x7f22580f0be0 msgr2=0x7f22580f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 -- 172.21.15.76:0/1366465490 shutdown_connections 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.445+0000 7f225ed0f700 1 -- 172.21.15.76:0/1366465490 wait complete. 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225ed0f700 1 Processor -- start 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225ed0f700 1 -- start start 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225ed0f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22581094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225ed0f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2258060320 con 0x7f22580f5480 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225caab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22581094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225caab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22581094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40106/0 (socket says 172.21.15.76:40106) 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225caab700 1 -- 172.21.15.76:0/705734665 learned_addr learned my addr 172.21.15.76:0/705734665 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225caab700 1 -- 172.21.15.76:0/705734665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f224c005040 con 0x7f22580f5480 2023-12-26T21:57:14.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.446+0000 7f225caab700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22581094f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f224c0089f0 tx=0x7f224c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:14.449 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.447+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f224c009320 con 0x7f22580f5480 2023-12-26T21:57:14.449 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.447+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2258060520 con 0x7f22580f5480 2023-12-26T21:57:14.449 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.447+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f224c013ec0 con 0x7f22580f5480 2023-12-26T21:57:14.449 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.447+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f224c014d30 con 0x7f22580f5480 2023-12-26T21:57:14.449 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.447+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2258109ba0 con 0x7f22580f5480 2023-12-26T21:57:14.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.447+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f224c014420 con 0x7f22580f5480 2023-12-26T21:57:14.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.448+0000 7f22557fa700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2248031090 0x7f2248033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:14.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.448+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f224c0424c0 con 0x7f22580f5480 2023-12-26T21:57:14.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.448+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2258045bb0 con 0x7f22580f5480 2023-12-26T21:57:14.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.448+0000 7f2257fff700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2248031090 0x7f2248033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:14.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.448+0000 7f2257fff700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2248031090 0x7f2248033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f22440096a0 tx=0x7f2244006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:14.453 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.453+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f224c014750 con 0x7f22580f5480 2023-12-26T21:57:14.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.662+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f225810a360 con 0x7f22580f5480 2023-12-26T21:57:14.664 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.663+0000 7f22557fa700 1 -- 172.21.15.76:0/705734665 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f224c01d380 con 0x7f22580f5480 2023-12-26T21:57:14.664 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:14.664 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:14.665 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2248031090 msgr2=0x7f2248033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:14.665 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2248031090 0x7f2248033550 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f22440096a0 tx=0x7f2244006b40).stop 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 msgr2=0x7f22581094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22581094f0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f224c0089f0 tx=0x7f224c00b040).stop 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 shutdown_connections 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2248031090 0x7f2248033550 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 --2- 172.21.15.76:0/705734665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f22580f5480 0x7f22581094f0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 >> 172.21.15.76:0/705734665 conn(0x7f22580f0be0 msgr2=0x7f22580f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 shutdown_connections 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:14.665+0000 7f225ed0f700 1 -- 172.21.15.76:0/705734665 wait complete. 2023-12-26T21:57:14.666 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:15.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:13.325896+0000 mgr.smithi059.gwkiyc (mgr.14156) 16 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:15.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:14.664755+0000 mon.smithi059 (mon.0) 109 : audit [DBG] from='client.? 172.21.15.76:0/705734665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:16.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:16.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:17.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:15.326152+0000 mgr.smithi059.gwkiyc (mgr.14156) 17 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 -- 172.21.15.76:0/4289291085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 msgr2=0x7fbb1c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 --2- 172.21.15.76:0/4289291085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c0f58a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fbb10009fb0 tx=0x7fbb10009b30).stop 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 -- 172.21.15.76:0/4289291085 shutdown_connections 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 --2- 172.21.15.76:0/4289291085 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c0f58a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 -- 172.21.15.76:0/4289291085 >> 172.21.15.76:0/4289291085 conn(0x7fbb1c0f0be0 msgr2=0x7fbb1c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 -- 172.21.15.76:0/4289291085 shutdown_connections 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 -- 172.21.15.76:0/4289291085 wait complete. 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.254+0000 7fbb222c5700 1 Processor -- start 2023-12-26T21:57:17.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb222c5700 1 -- start start 2023-12-26T21:57:17.256 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb222c5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb222c5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb1c105be0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb1bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb1bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60098/0 (socket says 172.21.15.76:60098) 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb1bfff700 1 -- 172.21.15.76:0/2630379868 learned_addr learned my addr 172.21.15.76:0/2630379868 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb1bfff700 1 -- 172.21.15.76:0/2630379868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb10005040 con 0x7fbb1c0f5480 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.255+0000 7fbb1bfff700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fbb100089f0 tx=0x7fbb1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.256+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb10009320 con 0x7fbb1c0f5480 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.256+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbb10013ec0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.256+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb1c105de0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.256+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb10014d30 con 0x7fbb1c0f5480 2023-12-26T21:57:17.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.256+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb1c106270 con 0x7fbb1c0f5480 2023-12-26T21:57:17.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.257+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbb100095f0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.257+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb1c045bb0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.257+0000 7fbb18ff9700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb04031090 0x7fbb04033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:17.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.257+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbb100424e0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.257+0000 7fbb1b7fe700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb04031090 0x7fbb04033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:17.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.258+0000 7fbb1b7fe700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb04031090 0x7fbb04033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fbb0c0096a0 tx=0x7fbb0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:17.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.262+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbb1001d7f0 con 0x7fbb1c0f5480 2023-12-26T21:57:17.469 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.468+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbb1c0faf70 con 0x7fbb1c0f5480 2023-12-26T21:57:17.469 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.468+0000 7fbb18ff9700 1 -- 172.21.15.76:0/2630379868 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbb1000fe40 con 0x7fbb1c0f5480 2023-12-26T21:57:17.469 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:17.470 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb04031090 msgr2=0x7fbb04033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:17.470 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb04031090 0x7fbb04033550 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fbb0c0096a0 tx=0x7fbb0c006b40).stop 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 msgr2=0x7fbb1c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fbb100089f0 tx=0x7fbb1000b040).stop 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 shutdown_connections 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb04031090 0x7fbb04033550 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 --2- 172.21.15.76:0/2630379868 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbb1c0f5480 0x7fbb1c1056f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 >> 172.21.15.76:0/2630379868 conn(0x7fbb1c0f0be0 msgr2=0x7fbb1c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 shutdown_connections 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:17.470+0000 7fbb222c5700 1 -- 172.21.15.76:0/2630379868 wait complete. 2023-12-26T21:57:17.471 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:18.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:17 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:17.470043+0000 mon.smithi059 (mon.0) 110 : audit [DBG] from='client.? 172.21.15.76:0/2630379868' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:18.866 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:18.866 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:19.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:17.326498+0000 mgr.smithi059.gwkiyc (mgr.14156) 18 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 -- 172.21.15.76:0/33228560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 msgr2=0x7fe3000f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 --2- 172.21.15.76:0/33228560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3000f58d0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fe2f4009fb0 tx=0x7fe2f4009b30).stop 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 -- 172.21.15.76:0/33228560 shutdown_connections 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 --2- 172.21.15.76:0/33228560 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3000f58d0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 -- 172.21.15.76:0/33228560 >> 172.21.15.76:0/33228560 conn(0x7fe3000f0c10 msgr2=0x7fe3000f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 -- 172.21.15.76:0/33228560 shutdown_connections 2023-12-26T21:57:20.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.171+0000 7fe305192700 1 -- 172.21.15.76:0/33228560 wait complete. 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe305192700 1 Processor -- start 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe305192700 1 -- start start 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe305192700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe300109500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe305192700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300060320 con 0x7fe3000f54b0 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe2fed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe300109500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe2fed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe300109500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60114/0 (socket says 172.21.15.76:60114) 2023-12-26T21:57:20.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.172+0000 7fe2fed9d700 1 -- 172.21.15.76:0/2687028309 learned_addr learned my addr 172.21.15.76:0/2687028309 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe2fed9d700 1 -- 172.21.15.76:0/2687028309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2f4005040 con 0x7fe3000f54b0 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe2fed9d700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe300109500 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe2f40089f0 tx=0x7fe2f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f4009320 con 0x7fe3000f54b0 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2f4013ec0 con 0x7fe3000f54b0 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe300060520 con 0x7fe3000f54b0 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2f4014d30 con 0x7fe3000f54b0 2023-12-26T21:57:20.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.173+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe300109bb0 con 0x7fe3000f54b0 2023-12-26T21:57:20.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.174+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe2f40095f0 con 0x7fe3000f54b0 2023-12-26T21:57:20.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.174+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300045bb0 con 0x7fe3000f54b0 2023-12-26T21:57:20.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.174+0000 7fe2e77fe700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2e8031090 0x7fe2e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:20.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.174+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2f40423e0 con 0x7fe3000f54b0 2023-12-26T21:57:20.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.174+0000 7fe2fe59c700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2e8031090 0x7fe2e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:20.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.175+0000 7fe2fe59c700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2e8031090 0x7fe2e8033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe2f00096a0 tx=0x7fe2f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:20.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.180+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe2f401d7f0 con 0x7fe3000f54b0 2023-12-26T21:57:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.383+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe30010a3d0 con 0x7fe3000f54b0 2023-12-26T21:57:20.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.384+0000 7fe2e77fe700 1 -- 172.21.15.76:0/2687028309 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2f401d7f0 con 0x7fe3000f54b0 2023-12-26T21:57:20.385 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:20.385 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2e8031090 msgr2=0x7fe2e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2e8031090 0x7fe2e8033550 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fe2f00096a0 tx=0x7fe2f0006b40).stop 2023-12-26T21:57:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 msgr2=0x7fe300109500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe300109500 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fe2f40089f0 tx=0x7fe2f400b040).stop 2023-12-26T21:57:20.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 shutdown_connections 2023-12-26T21:57:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2e8031090 0x7fe2e8033550 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.385+0000 7fe305192700 1 --2- 172.21.15.76:0/2687028309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe300109500 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.386+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 >> 172.21.15.76:0/2687028309 conn(0x7fe3000f0c10 msgr2=0x7fe3000f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.386+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 shutdown_connections 2023-12-26T21:57:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:20.386+0000 7fe305192700 1 -- 172.21.15.76:0/2687028309 wait complete. 2023-12-26T21:57:20.387 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:21.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:19.326771+0000 mgr.smithi059.gwkiyc (mgr.14156) 19 : cluster [DBG] pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:21.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:20.385361+0000 mon.smithi059 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.76:0/2687028309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:21.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:21.773 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- 172.21.15.76:0/2349307964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 msgr2=0x7fecf00f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 --2- 172.21.15.76:0/2349307964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f9200 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fece0009fb0 tx=0x7fece0009b30).stop 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- 172.21.15.76:0/2349307964 shutdown_connections 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 --2- 172.21.15.76:0/2349307964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f9200 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- 172.21.15.76:0/2349307964 >> 172.21.15.76:0/2349307964 conn(0x7fecf00f0be0 msgr2=0x7fecf00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- 172.21.15.76:0/2349307964 shutdown_connections 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- 172.21.15.76:0/2349307964 wait complete. 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 Processor -- start 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- start start 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:22.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.993+0000 7fecf569e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fecf00f5100 con 0x7fecf00602c0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7feceeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7feceeffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60136/0 (socket says 172.21.15.76:60136) 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7feceeffd700 1 -- 172.21.15.76:0/3184751598 learned_addr learned my addr 172.21.15.76:0/3184751598 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7feceeffd700 1 -- 172.21.15.76:0/3184751598 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece0005040 con 0x7fecf00602c0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7feceeffd700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f6c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fece0009d00 tx=0x7fece0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fece0018070 con 0x7fecf00602c0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fece001d420 con 0x7fecf00602c0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fecf00f52a0 con 0x7fecf00602c0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fece001c5a0 con 0x7fecf00602c0 2023-12-26T21:57:22.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.994+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fecf00f5730 con 0x7fecf00602c0 2023-12-26T21:57:22.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.995+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fece0007a90 con 0x7fecf00602c0 2023-12-26T21:57:22.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.995+0000 7fecd77fe700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fecd8031090 0x7fecd8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:22.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.995+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fecf00591e0 con 0x7fecf00602c0 2023-12-26T21:57:22.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.995+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fece0042870 con 0x7fecf00602c0 2023-12-26T21:57:22.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.996+0000 7fecee7fc700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fecd8031090 0x7fecd8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:22.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:22.996+0000 7fecee7fc700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fecd8031090 0x7fecd8033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fece40096a0 tx=0x7fece4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:23.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.000+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fece0027440 con 0x7fecf00602c0 2023-12-26T21:57:23.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:21.327125+0000 mgr.smithi059.gwkiyc (mgr.14156) 20 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:23.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.206+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fecf00fc070 con 0x7fecf00602c0 2023-12-26T21:57:23.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.207+0000 7fecd77fe700 1 -- 172.21.15.76:0/3184751598 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fece001b370 con 0x7fecf00602c0 2023-12-26T21:57:23.207 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:23.207 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.208+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fecd8031090 msgr2=0x7fecd8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.208+0000 7fecf569e700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fecd8031090 0x7fecd8033550 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fece40096a0 tx=0x7fece4006b40).stop 2023-12-26T21:57:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.208+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 msgr2=0x7fecf00f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.208+0000 7fecf569e700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f6c80 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fece0009d00 tx=0x7fece0009b30).stop 2023-12-26T21:57:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.209+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 shutdown_connections 2023-12-26T21:57:23.208 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.209+0000 7fecf569e700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fecd8031090 0x7fecd8033550 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:23.209 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.209+0000 7fecf569e700 1 --2- 172.21.15.76:0/3184751598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fecf00602c0 0x7fecf00f6c80 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:23.209 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.209+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 >> 172.21.15.76:0/3184751598 conn(0x7fecf00f0be0 msgr2=0x7fecf00f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:23.209 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.209+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 shutdown_connections 2023-12-26T21:57:23.209 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:23.209+0000 7fecf569e700 1 -- 172.21.15.76:0/3184751598 wait complete. 2023-12-26T21:57:23.209 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:24.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:23.207685+0000 mon.smithi059 (mon.0) 112 : audit [DBG] from='client.? 172.21.15.76:0/3184751598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:24.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:24.586 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:25.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:23.327469+0000 mgr.smithi059.gwkiyc (mgr.14156) 21 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:25.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 -- 172.21.15.76:0/3295999311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 msgr2=0x7fe2e00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:25.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 --2- 172.21.15.76:0/3295999311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e00f9160 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fe2c8009fb0 tx=0x7fe2c8009b30).stop 2023-12-26T21:57:25.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 -- 172.21.15.76:0/3295999311 shutdown_connections 2023-12-26T21:57:25.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 --2- 172.21.15.76:0/3295999311 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e00f9160 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:25.805 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 -- 172.21.15.76:0/3295999311 >> 172.21.15.76:0/3295999311 conn(0x7fe2e00f0be0 msgr2=0x7fe2e00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:25.805 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 -- 172.21.15.76:0/3295999311 shutdown_connections 2023-12-26T21:57:25.805 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.804+0000 7fe2e5506700 1 -- 172.21.15.76:0/3295999311 wait complete. 2023-12-26T21:57:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2e5506700 1 Processor -- start 2023-12-26T21:57:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2e5506700 1 -- start start 2023-12-26T21:57:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2e5506700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e0106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2e5506700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2e0106c50 con 0x7fe2e0060360 2023-12-26T21:57:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2deffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e0106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:25.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2deffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e0106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60738/0 (socket says 172.21.15.76:60738) 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.805+0000 7fe2deffd700 1 -- 172.21.15.76:0/1737594727 learned_addr learned my addr 172.21.15.76:0/1737594727 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2deffd700 1 -- 172.21.15.76:0/1737594727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2c8005040 con 0x7fe2e0060360 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2deffd700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e0106760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe2c8009f80 tx=0x7fe2c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2c8009320 con 0x7fe2e0060360 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe2e0106e50 con 0x7fe2e0060360 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2c8013680 con 0x7fe2e0060360 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2c801eda0 con 0x7fe2e0060360 2023-12-26T21:57:25.807 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.806+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2e01072e0 con 0x7fe2e0060360 2023-12-26T21:57:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.807+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe2c801e420 con 0x7fe2e0060360 2023-12-26T21:57:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.807+0000 7fe2d77fe700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2cc031090 0x7fe2cc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.807+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe2e0045bb0 con 0x7fe2e0060360 2023-12-26T21:57:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.807+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2c80422f0 con 0x7fe2e0060360 2023-12-26T21:57:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.808+0000 7fe2de7fc700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2cc031090 0x7fe2cc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:25.808 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.808+0000 7fe2de7fc700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2cc031090 0x7fe2cc033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe2d00096a0 tx=0x7fe2d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:25.812 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:25.813+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe2c801e700 con 0x7fe2e0060360 2023-12-26T21:57:26.018 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.018+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe2e01080e0 con 0x7fe2e0060360 2023-12-26T21:57:26.018 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.019+0000 7fe2d77fe700 1 -- 172.21.15.76:0/1737594727 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2c801cb10 con 0x7fe2e0060360 2023-12-26T21:57:26.019 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:26.019 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2cc031090 msgr2=0x7fe2cc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2cc031090 0x7fe2cc033550 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7fe2d00096a0 tx=0x7fe2d0006b40).stop 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 msgr2=0x7fe2e0106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e0106760 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fe2c8009f80 tx=0x7fe2c800b040).stop 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 shutdown_connections 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2cc031090 0x7fe2cc033550 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 --2- 172.21.15.76:0/1737594727 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe2e0060360 0x7fe2e0106760 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 >> 172.21.15.76:0/1737594727 conn(0x7fe2e00f0be0 msgr2=0x7fe2e00f16c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 shutdown_connections 2023-12-26T21:57:26.020 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:26.020+0000 7fe2e5506700 1 -- 172.21.15.76:0/1737594727 wait complete. 2023-12-26T21:57:26.021 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:27.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:25.327743+0000 mgr.smithi059.gwkiyc (mgr.14156) 22 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:27.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:26.019282+0000 mon.smithi059 (mon.0) 113 : audit [DBG] from='client.? 172.21.15.76:0/1737594727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:27.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:27.415 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 -- 172.21.15.76:0/271769775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 msgr2=0x7fc4a00f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 --2- 172.21.15.76:0/271769775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a00f8fb0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fc490009fb0 tx=0x7fc490009b30).stop 2023-12-26T21:57:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 -- 172.21.15.76:0/271769775 shutdown_connections 2023-12-26T21:57:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 --2- 172.21.15.76:0/271769775 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a00f8fb0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 -- 172.21.15.76:0/271769775 >> 172.21.15.76:0/271769775 conn(0x7fc4a00f0a30 msgr2=0x7fc4a00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:28.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 -- 172.21.15.76:0/271769775 shutdown_connections 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.614+0000 7fc49ffff700 1 -- 172.21.15.76:0/271769775 wait complete. 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.615+0000 7fc49ffff700 1 Processor -- start 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.615+0000 7fc49ffff700 1 -- start start 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.615+0000 7fc49ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a01043a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.615+0000 7fc49ffff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4a0104890 con 0x7fc4a0060360 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a01043a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a01043a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60754/0 (socket says 172.21.15.76:60754) 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49effd700 1 -- 172.21.15.76:0/3663842417 learned_addr learned my addr 172.21.15.76:0/3663842417 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:28.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49effd700 1 -- 172.21.15.76:0/3663842417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc490005040 con 0x7fc4a0060360 2023-12-26T21:57:28.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49effd700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a01043a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc490009f80 tx=0x7fc49000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc490009320 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc490013680 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4a0104a90 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc49001eda0 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.616+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4a0104f20 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.617+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc490014420 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.617+0000 7fc4877fe700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc4880354e0 0x7fc4880379a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.617+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc4900423c0 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.617+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4a0045bb0 con 0x7fc4a0060360 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.617+0000 7fc49e7fc700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc4880354e0 0x7fc4880379a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:28.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.618+0000 7fc49e7fc700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc4880354e0 0x7fc4880379a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc4940096a0 tx=0x7fc494006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:28.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.622+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc49001c910 con 0x7fc4a0060360 2023-12-26T21:57:28.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.833+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4a00f9db0 con 0x7fc4a0060360 2023-12-26T21:57:28.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.834+0000 7fc4877fe700 1 -- 172.21.15.76:0/3663842417 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc49001c320 con 0x7fc4a0060360 2023-12-26T21:57:28.833 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:28.833 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc4880354e0 msgr2=0x7fc4880379a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc4880354e0 0x7fc4880379a0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7fc4940096a0 tx=0x7fc494006b40).stop 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 msgr2=0x7fc4a01043a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a01043a0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fc490009f80 tx=0x7fc49000b040).stop 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 shutdown_connections 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc4880354e0 0x7fc4880379a0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 --2- 172.21.15.76:0/3663842417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc4a0060360 0x7fc4a01043a0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 >> 172.21.15.76:0/3663842417 conn(0x7fc4a00f0a30 msgr2=0x7fc4a00f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 shutdown_connections 2023-12-26T21:57:28.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:28.835+0000 7fc49ffff700 1 -- 172.21.15.76:0/3663842417 wait complete. 2023-12-26T21:57:28.836 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:29.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:27.328048+0000 mgr.smithi059.gwkiyc (mgr.14156) 23 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:30.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:29 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:28.834094+0000 mon.smithi059 (mon.0) 114 : audit [DBG] from='client.? 172.21.15.76:0/3663842417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:30.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:30.531 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:31.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:29.328383+0000 mgr.smithi059.gwkiyc (mgr.14156) 24 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:31.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.738+0000 7f79464a6700 1 -- 172.21.15.76:0/2359062507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 msgr2=0x7f79400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.738+0000 7f79464a6700 1 --2- 172.21.15.76:0/2359062507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400f58a0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f7930009fb0 tx=0x7f7930009b30).stop 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.738+0000 7f79464a6700 1 -- 172.21.15.76:0/2359062507 shutdown_connections 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.738+0000 7f79464a6700 1 --2- 172.21.15.76:0/2359062507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400f58a0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.738+0000 7f79464a6700 1 -- 172.21.15.76:0/2359062507 >> 172.21.15.76:0/2359062507 conn(0x7f79400f0be0 msgr2=0x7f79400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.739+0000 7f79464a6700 1 -- 172.21.15.76:0/2359062507 shutdown_connections 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.739+0000 7f79464a6700 1 -- 172.21.15.76:0/2359062507 wait complete. 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.739+0000 7f79464a6700 1 Processor -- start 2023-12-26T21:57:31.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.739+0000 7f79464a6700 1 -- start start 2023-12-26T21:57:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.739+0000 7f79464a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:31.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.739+0000 7f79464a6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79400fb8e0 con 0x7f79400f5480 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.740+0000 7f793ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.740+0000 7f793ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60774/0 (socket says 172.21.15.76:60774) 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.740+0000 7f793ffff700 1 -- 172.21.15.76:0/314232468 learned_addr learned my addr 172.21.15.76:0/314232468 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.740+0000 7f793ffff700 1 -- 172.21.15.76:0/314232468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7930005040 con 0x7f79400f5480 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.740+0000 7f793ffff700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400fd460 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f79300089f0 tx=0x7f793000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.741+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930009320 con 0x7f79400f5480 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.741+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79400fba80 con 0x7f79400f5480 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.741+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7930013ec0 con 0x7f79400f5480 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.741+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7930014d30 con 0x7f79400f5480 2023-12-26T21:57:31.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.741+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79400fbf10 con 0x7f79400f5480 2023-12-26T21:57:31.742 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.742+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79400591e0 con 0x7f79400f5480 2023-12-26T21:57:31.743 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.743+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f79300095f0 con 0x7f79400f5480 2023-12-26T21:57:31.743 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.743+0000 7f793cff9700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7928031090 0x7f7928033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:31.743 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.743+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f79300425f0 con 0x7f79400f5480 2023-12-26T21:57:31.743 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.744+0000 7f793f7fe700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7928031090 0x7f7928033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:31.744 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.744+0000 7f793f7fe700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7928031090 0x7f7928033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f79340096a0 tx=0x7f7934006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:31.747 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.747+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f793000f990 con 0x7f79400f5480 2023-12-26T21:57:31.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.957+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79400faf70 con 0x7f79400f5480 2023-12-26T21:57:31.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.958+0000 7f793cff9700 1 -- 172.21.15.76:0/314232468 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f793000f340 con 0x7f79400f5480 2023-12-26T21:57:31.957 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:31.957 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7928031090 msgr2=0x7f7928033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7928031090 0x7f7928033550 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f79340096a0 tx=0x7f7934006b40).stop 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 msgr2=0x7f79400fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400fd460 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f79300089f0 tx=0x7f793000b040).stop 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 shutdown_connections 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7928031090 0x7f7928033550 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 --2- 172.21.15.76:0/314232468 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f79400f5480 0x7f79400fd460 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:31.960 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 >> 172.21.15.76:0/314232468 conn(0x7f79400f0be0 msgr2=0x7f794005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:31.961 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 shutdown_connections 2023-12-26T21:57:31.961 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:31.959+0000 7f79464a6700 1 -- 172.21.15.76:0/314232468 wait complete. 2023-12-26T21:57:31.961 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:33.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:31.328619+0000 mgr.smithi059.gwkiyc (mgr.14156) 25 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:33.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:31.958256+0000 mon.smithi059 (mon.0) 115 : audit [DBG] from='client.? 172.21.15.76:0/314232468' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:33.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:33.364 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 -- 172.21.15.76:0/1459678870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 msgr2=0x7f8ce40f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 --2- 172.21.15.76:0/1459678870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40f5e80 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f8cd8009fb0 tx=0x7f8cd8009b30).stop 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 -- 172.21.15.76:0/1459678870 shutdown_connections 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 --2- 172.21.15.76:0/1459678870 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40f5e80 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 -- 172.21.15.76:0/1459678870 >> 172.21.15.76:0/1459678870 conn(0x7f8ce40f0390 msgr2=0x7f8ce40f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 -- 172.21.15.76:0/1459678870 shutdown_connections 2023-12-26T21:57:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.572+0000 7f8ce9336700 1 -- 172.21.15.76:0/1459678870 wait complete. 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce9336700 1 Processor -- start 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce9336700 1 -- start start 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce9336700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40fd470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce9336700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce40fb8f0 con 0x7f8ce40f5a60 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40fd470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce2ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40fd470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60788/0 (socket says 172.21.15.76:60788) 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce2ffd700 1 -- 172.21.15.76:0/1193943956 learned_addr learned my addr 172.21.15.76:0/1193943956 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce2ffd700 1 -- 172.21.15.76:0/1193943956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8005040 con 0x7f8ce40f5a60 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.573+0000 7f8ce2ffd700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40fd470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8cd8000c00 tx=0x7f8cd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.574+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd80095f0 con 0x7f8ce40f5a60 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.574+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8cd8007e50 con 0x7f8ce40f5a60 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.574+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce40fba90 con 0x7f8ce40f5a60 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.574+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd8014e70 con 0x7f8ce40f5a60 2023-12-26T21:57:34.575 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.574+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce40fbf20 con 0x7f8ce40f5a60 2023-12-26T21:57:34.576 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.574+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8cd8025070 con 0x7f8ce40f5a60 2023-12-26T21:57:34.576 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.575+0000 7f8ccb7fe700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8ccc031090 0x7f8ccc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:34.576 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.575+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8cd8041d20 con 0x7f8ce40f5a60 2023-12-26T21:57:34.576 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.575+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce40591e0 con 0x7f8ce40f5a60 2023-12-26T21:57:34.577 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.575+0000 7f8ce27fc700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8ccc031090 0x7f8ccc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:34.577 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.575+0000 7f8ce27fc700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8ccc031090 0x7f8ccc033550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f8cd40096a0 tx=0x7f8cd4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:34.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.580+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8cd80079c0 con 0x7f8ce40f5a60 2023-12-26T21:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.788+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ce40faf80 con 0x7f8ce40f5a60 2023-12-26T21:57:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.789+0000 7f8ccb7fe700 1 -- 172.21.15.76:0/1193943956 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cd800f300 con 0x7f8ce40f5a60 2023-12-26T21:57:34.790 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:34.790 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8ccc031090 msgr2=0x7f8ccc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:34.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8ccc031090 0x7f8ccc033550 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f8cd40096a0 tx=0x7f8cd4006b40).stop 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 msgr2=0x7f8ce40fd470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40fd470 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f8cd8000c00 tx=0x7f8cd800b040).stop 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 shutdown_connections 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8ccc031090 0x7f8ccc033550 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 --2- 172.21.15.76:0/1193943956 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f5a60 0x7f8ce40fd470 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 >> 172.21.15.76:0/1193943956 conn(0x7f8ce40f0390 msgr2=0x7f8ce40f5020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 shutdown_connections 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:34.791+0000 7f8ce9336700 1 -- 172.21.15.76:0/1193943956 wait complete. 2023-12-26T21:57:34.792 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:35.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:33.328899+0000 mgr.smithi059.gwkiyc (mgr.14156) 26 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:36.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:35 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:34.790672+0000 mon.smithi059 (mon.0) 116 : audit [DBG] from='client.? 172.21.15.76:0/1193943956' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:36.192 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:36.192 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:37.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:35.329157+0000 mgr.smithi059.gwkiyc (mgr.14156) 27 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 -- 172.21.15.76:0/2346796748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 msgr2=0x7f86000f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 --2- 172.21.15.76:0/2346796748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86000f4c60 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f85f4009fb0 tx=0x7f85f4009b30).stop 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 -- 172.21.15.76:0/2346796748 shutdown_connections 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 --2- 172.21.15.76:0/2346796748 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86000f4c60 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 -- 172.21.15.76:0/2346796748 >> 172.21.15.76:0/2346796748 conn(0x7f86000f0360 msgr2=0x7f86000f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 -- 172.21.15.76:0/2346796748 shutdown_connections 2023-12-26T21:57:37.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 -- 172.21.15.76:0/2346796748 wait complete. 2023-12-26T21:57:37.370 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.368+0000 7f86067a9700 1 Processor -- start 2023-12-26T21:57:37.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f86067a9700 1 -- start start 2023-12-26T21:57:37.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f86067a9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86001067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:37.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f86067a9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600106cb0 con 0x7f86000f4840 2023-12-26T21:57:37.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f85fffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86001067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f85fffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86001067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:43438/0 (socket says 172.21.15.76:43438) 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f85fffff700 1 -- 172.21.15.76:0/1212719164 learned_addr learned my addr 172.21.15.76:0/1212719164 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f85fffff700 1 -- 172.21.15.76:0/1212719164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4005040 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f85fffff700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86001067c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f85f40089f0 tx=0x7f85f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.369+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85f4009320 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.370+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f85f4013ec0 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.370+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85f4014d30 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.370+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600106eb0 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.370+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8600107340 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.370+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f85f40095f0 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.371+0000 7f85fcff9700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f85e80310e0 0x7f85e80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.371+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f85f4042460 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.371+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8600045bb0 con 0x7f86000f4840 2023-12-26T21:57:37.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.371+0000 7f85ff7fe700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f85e80310e0 0x7f85e80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:37.373 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.371+0000 7f85ff7fe700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f85e80310e0 0x7f85e80335a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f85f00096a0 tx=0x7f85f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:37.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.376+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f85f401d7f0 con 0x7f86000f4840 2023-12-26T21:57:37.583 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.583+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8600108140 con 0x7f86000f4840 2023-12-26T21:57:37.584 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.583+0000 7f85fcff9700 1 -- 172.21.15.76:0/1212719164 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f85f400fe40 con 0x7f86000f4840 2023-12-26T21:57:37.584 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:37.584 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f85e80310e0 msgr2=0x7f85e80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f85e80310e0 0x7f85e80335a0 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f85f00096a0 tx=0x7f85f0006b40).stop 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 msgr2=0x7f86001067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86001067c0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f85f40089f0 tx=0x7f85f400b040).stop 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 shutdown_connections 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f85e80310e0 0x7f85e80335a0 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:37.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 --2- 172.21.15.76:0/1212719164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f86000f4840 0x7f86001067c0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:37.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 >> 172.21.15.76:0/1212719164 conn(0x7f86000f0360 msgr2=0x7f86000f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:37.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 shutdown_connections 2023-12-26T21:57:37.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:37.585+0000 7f86067a9700 1 -- 172.21.15.76:0/1212719164 wait complete. 2023-12-26T21:57:37.587 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:38.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:37 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:37.585064+0000 mon.smithi059 (mon.0) 117 : audit [DBG] from='client.? 172.21.15.76:0/1212719164' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:38.977 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:38.978 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:39.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:37.329538+0000 mgr.smithi059.gwkiyc (mgr.14156) 28 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:40.124 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 -- 172.21.15.76:0/2594906020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 msgr2=0x7efef00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:40.124 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 --2- 172.21.15.76:0/2594906020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef00f58a0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7efee0009fb0 tx=0x7efee0009b30).stop 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 -- 172.21.15.76:0/2594906020 shutdown_connections 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 --2- 172.21.15.76:0/2594906020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef00f58a0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 -- 172.21.15.76:0/2594906020 >> 172.21.15.76:0/2594906020 conn(0x7efef00f0be0 msgr2=0x7efef00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 -- 172.21.15.76:0/2594906020 shutdown_connections 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.121+0000 7efef6533700 1 -- 172.21.15.76:0/2594906020 wait complete. 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efef6533700 1 Processor -- start 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efef6533700 1 -- start start 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efef6533700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efef6533700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efef0060300 con 0x7efef00f5480 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efeeffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:40.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efeeffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:43458/0 (socket says 172.21.15.76:43458) 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efeeffff700 1 -- 172.21.15.76:0/40145240 learned_addr learned my addr 172.21.15.76:0/40145240 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.122+0000 7efeeffff700 1 -- 172.21.15.76:0/40145240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efee0005040 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.123+0000 7efeeffff700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef0109510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7efee0000c00 tx=0x7efee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.123+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee00095f0 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.123+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efef0060500 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.123+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efee0007e50 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.123+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efee0014e70 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.123+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efef0109bc0 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.124+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efee0025070 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.124+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efef00591e0 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.124+0000 7efeecff9700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efed80310e0 0x7efed80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.124+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efee0041d30 con 0x7efef00f5480 2023-12-26T21:57:40.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.124+0000 7efeef7fe700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efed80310e0 0x7efed80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:40.127 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.124+0000 7efeef7fe700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efed80310e0 0x7efed80335a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:40.129 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.129+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efee00079c0 con 0x7efef00f5480 2023-12-26T21:57:40.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.339+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efef0045bb0 con 0x7efef00f5480 2023-12-26T21:57:40.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.340+0000 7efeecff9700 1 -- 172.21.15.76:0/40145240 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efee000f300 con 0x7efef00f5480 2023-12-26T21:57:40.341 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:40.341 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:40.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efed80310e0 msgr2=0x7efed80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:40.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efed80310e0 0x7efed80335a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7efee40096a0 tx=0x7efee4006b40).stop 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 msgr2=0x7efef0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef0109510 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7efee0000c00 tx=0x7efee000b040).stop 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 shutdown_connections 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efed80310e0 0x7efed80335a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 --2- 172.21.15.76:0/40145240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efef00f5480 0x7efef0109510 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 >> 172.21.15.76:0/40145240 conn(0x7efef00f0be0 msgr2=0x7efef00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 shutdown_connections 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:40.342+0000 7efef6533700 1 -- 172.21.15.76:0/40145240 wait complete. 2023-12-26T21:57:40.343 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:41.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:39.329906+0000 mgr.smithi059.gwkiyc (mgr.14156) 29 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:41.046 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:40.341472+0000 mon.smithi059 (mon.0) 118 : audit [DBG] from='client.? 172.21.15.76:0/40145240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:41.730 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:41.730 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:42.962 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 -- 172.21.15.76:0/366112731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 msgr2=0x7f58f40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:42.962 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 --2- 172.21.15.76:0/366112731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f40f90f0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f58e4009fb0 tx=0x7f58e4009b30).stop 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 -- 172.21.15.76:0/366112731 shutdown_connections 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 --2- 172.21.15.76:0/366112731 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f40f90f0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 -- 172.21.15.76:0/366112731 >> 172.21.15.76:0/366112731 conn(0x7f58f40f0b70 msgr2=0x7f58f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 -- 172.21.15.76:0/366112731 shutdown_connections 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.961+0000 7f58fb6f8700 1 -- 172.21.15.76:0/366112731 wait complete. 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58fb6f8700 1 Processor -- start 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58fb6f8700 1 -- start start 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58fb6f8700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f4106a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58fb6f8700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58f4106f10 con 0x7f58f4060360 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58f9494700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f4106a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58f9494700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f4106a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:43488/0 (socket says 172.21.15.76:43488) 2023-12-26T21:57:42.963 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58f9494700 1 -- 172.21.15.76:0/3340943154 learned_addr learned my addr 172.21.15.76:0/3340943154 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58f9494700 1 -- 172.21.15.76:0/3340943154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58e4005040 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.962+0000 7f58f9494700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f4106a20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f58e4009f80 tx=0x7f58e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.963+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e4009320 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.963+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f58e4013680 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.963+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58e401eda0 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.963+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58f4107110 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.963+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58f41075a0 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.964+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f58e401e420 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.964+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58f4045bb0 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.964+0000 7f58e9ffb700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f58e00310e0 0x7f58e00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.964+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f58e4042350 con 0x7f58f4060360 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.964+0000 7f58f8c93700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f58e00310e0 0x7f58e00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:42.965 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.964+0000 7f58f8c93700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f58e00310e0 0x7f58e00335a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f58f00096a0 tx=0x7f58f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:42.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:42.969+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f58e401e6b0 con 0x7f58f4060360 2023-12-26T21:57:43.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:41.330313+0000 mgr.smithi059.gwkiyc (mgr.14156) 30 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:43.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.174+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58f405eb30 con 0x7f58f4060360 2023-12-26T21:57:43.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.175+0000 7f58e9ffb700 1 -- 172.21.15.76:0/3340943154 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58e4004490 con 0x7f58f4060360 2023-12-26T21:57:43.175 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:43.176 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:43.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f58e00310e0 msgr2=0x7f58e00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:43.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f58e00310e0 0x7f58e00335a0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f58f00096a0 tx=0x7f58f0006b40).stop 2023-12-26T21:57:43.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 msgr2=0x7f58f4106a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:43.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f4106a20 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f58e4009f80 tx=0x7f58e400b040).stop 2023-12-26T21:57:43.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 shutdown_connections 2023-12-26T21:57:43.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f58e00310e0 0x7f58e00335a0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:43.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 --2- 172.21.15.76:0/3340943154 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58f4060360 0x7f58f4106a20 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:43.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.176+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 >> 172.21.15.76:0/3340943154 conn(0x7f58f40f0b70 msgr2=0x7f58f40f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:43.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.177+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 shutdown_connections 2023-12-26T21:57:43.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:43.177+0000 7f58fb6f8700 1 -- 172.21.15.76:0/3340943154 wait complete. 2023-12-26T21:57:43.178 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:44.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:43 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:43.176502+0000 mon.smithi059 (mon.0) 119 : audit [DBG] from='client.? 172.21.15.76:0/3340943154' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:44.571 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:44.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:45.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:43.330707+0000 mgr.smithi059.gwkiyc (mgr.14156) 31 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 -- 172.21.15.76:0/3205336412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 msgr2=0x7fc7940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 --2- 172.21.15.76:0/3205336412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc7940f58a0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fc784009fb0 tx=0x7fc784009b30).stop 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 -- 172.21.15.76:0/3205336412 shutdown_connections 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 --2- 172.21.15.76:0/3205336412 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc7940f58a0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 -- 172.21.15.76:0/3205336412 >> 172.21.15.76:0/3205336412 conn(0x7fc7940f0be0 msgr2=0x7fc7940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 -- 172.21.15.76:0/3205336412 shutdown_connections 2023-12-26T21:57:45.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.953+0000 7fc79a93d700 1 -- 172.21.15.76:0/3205336412 wait complete. 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc79a93d700 1 Processor -- start 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc79a93d700 1 -- start start 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc79a93d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc794109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc79a93d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc794060300 con 0x7fc7940f5480 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc793fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc794109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc793fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc794109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50040/0 (socket says 172.21.15.76:50040) 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc793fff700 1 -- 172.21.15.76:0/327876778 learned_addr learned my addr 172.21.15.76:0/327876778 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.954+0000 7fc793fff700 1 -- 172.21.15.76:0/327876778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc784005040 con 0x7fc7940f5480 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.955+0000 7fc793fff700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc794109510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc7840089f0 tx=0x7fc78400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.955+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc784009320 con 0x7fc7940f5480 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.955+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc794060500 con 0x7fc7940f5480 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.955+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc784013ec0 con 0x7fc7940f5480 2023-12-26T21:57:45.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.955+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc784014d30 con 0x7fc7940f5480 2023-12-26T21:57:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.955+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc794109bc0 con 0x7fc7940f5480 2023-12-26T21:57:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.956+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc784014420 con 0x7fc7940f5480 2023-12-26T21:57:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.956+0000 7fc790ff9700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc77c031090 0x7fc77c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:45.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.956+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7940591e0 con 0x7fc7940f5480 2023-12-26T21:57:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.956+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc784042400 con 0x7fc7940f5480 2023-12-26T21:57:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.956+0000 7fc7937fe700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc77c031090 0x7fc77c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:45.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.957+0000 7fc7937fe700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc77c031090 0x7fc77c033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc7880096a0 tx=0x7fc788006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:45.961 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:45.961+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc78401d380 con 0x7fc7940f5480 2023-12-26T21:57:46.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:45.465166+0000 mon.smithi059 (mon.0) 120 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T21:57:46.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:45.466092+0000 mon.smithi059 (mon.0) 121 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T21:57:46.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.167+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc794045bb0 con 0x7fc7940f5480 2023-12-26T21:57:46.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.168+0000 7fc790ff9700 1 -- 172.21.15.76:0/327876778 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc78400f340 con 0x7fc7940f5480 2023-12-26T21:57:46.169 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:46.169 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:46.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc77c031090 msgr2=0x7fc77c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:46.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc77c031090 0x7fc77c033550 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7fc7880096a0 tx=0x7fc788006b40).stop 2023-12-26T21:57:46.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 msgr2=0x7fc794109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:46.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc794109510 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fc7840089f0 tx=0x7fc78400b040).stop 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 shutdown_connections 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc77c031090 0x7fc77c033550 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 --2- 172.21.15.76:0/327876778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7940f5480 0x7fc794109510 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 >> 172.21.15.76:0/327876778 conn(0x7fc7940f0be0 msgr2=0x7fc7940f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 shutdown_connections 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:46.170+0000 7fc79a93d700 1 -- 172.21.15.76:0/327876778 wait complete. 2023-12-26T21:57:46.171 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:47.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:45.331087+0000 mgr.smithi059.gwkiyc (mgr.14156) 32 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:47.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:46.169788+0000 mon.smithi059 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.76:0/327876778' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:47.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:47.588 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 -- 172.21.15.76:0/2873928728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 msgr2=0x7fc6a00fb5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 --2- 172.21.15.76:0/2873928728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a00fb5e0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fc690009fb0 tx=0x7fc690009b30).stop 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 -- 172.21.15.76:0/2873928728 shutdown_connections 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 --2- 172.21.15.76:0/2873928728 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a00fb5e0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 -- 172.21.15.76:0/2873928728 >> 172.21.15.76:0/2873928728 conn(0x7fc6a00f0be0 msgr2=0x7fc6a00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 -- 172.21.15.76:0/2873928728 shutdown_connections 2023-12-26T21:57:48.766 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.765+0000 7fc6a787c700 1 -- 172.21.15.76:0/2873928728 wait complete. 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a787c700 1 Processor -- start 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a787c700 1 -- start start 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a787c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a0108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a787c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6a0108e60 con 0x7fc6a00602e0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a5618700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a0108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a5618700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a0108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50058/0 (socket says 172.21.15.76:50058) 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.766+0000 7fc6a5618700 1 -- 172.21.15.76:0/2570032756 learned_addr learned my addr 172.21.15.76:0/2570032756 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc6a5618700 1 -- 172.21.15.76:0/2570032756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc690005040 con 0x7fc6a00602e0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc6a5618700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a0108970 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc690009f80 tx=0x7fc69000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc690009320 con 0x7fc6a00602e0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6a0109060 con 0x7fc6a00602e0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc690013680 con 0x7fc6a00602e0 2023-12-26T21:57:48.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc69001eda0 con 0x7fc6a00602e0 2023-12-26T21:57:48.769 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.767+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6a01094f0 con 0x7fc6a00602e0 2023-12-26T21:57:48.769 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.768+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc69001e420 con 0x7fc6a00602e0 2023-12-26T21:57:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.768+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6a0045bb0 con 0x7fc6a00602e0 2023-12-26T21:57:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.768+0000 7fc695ffb700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc68c031090 0x7fc68c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.768+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc690042450 con 0x7fc6a00602e0 2023-12-26T21:57:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.768+0000 7fc6a4e17700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc68c031090 0x7fc68c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:48.770 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.769+0000 7fc6a4e17700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc68c031090 0x7fc68c033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc69c0096a0 tx=0x7fc69c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:48.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.773+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc69001d380 con 0x7fc6a00602e0 2023-12-26T21:57:48.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.981+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc6a010a3a0 con 0x7fc6a00602e0 2023-12-26T21:57:48.982 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.982+0000 7fc695ffb700 1 -- 172.21.15.76:0/2570032756 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc690004490 con 0x7fc6a00602e0 2023-12-26T21:57:48.982 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:48.983 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:48.984 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc68c031090 msgr2=0x7fc68c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:48.984 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc68c031090 0x7fc68c033550 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fc69c0096a0 tx=0x7fc69c006b40).stop 2023-12-26T21:57:48.984 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 msgr2=0x7fc6a0108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:48.984 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a0108970 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7fc690009f80 tx=0x7fc69000b040).stop 2023-12-26T21:57:48.984 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 shutdown_connections 2023-12-26T21:57:48.985 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc68c031090 0x7fc68c033550 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:48.985 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 --2- 172.21.15.76:0/2570032756 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc6a00602e0 0x7fc6a0108970 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:48.985 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 >> 172.21.15.76:0/2570032756 conn(0x7fc6a00f0be0 msgr2=0x7fc6a00f1690 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:48.985 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 shutdown_connections 2023-12-26T21:57:48.985 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:48.983+0000 7fc6a787c700 1 -- 172.21.15.76:0/2570032756 wait complete. 2023-12-26T21:57:48.985 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:49.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:47.331506+0000 mgr.smithi059.gwkiyc (mgr.14156) 33 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:50.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:49 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:48.983216+0000 mon.smithi059 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.76:0/2570032756' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:50.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:50.401 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:51.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:49.331827+0000 mgr.smithi059.gwkiyc (mgr.14156) 34 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:51.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 -- 172.21.15.76:0/4165371279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 msgr2=0x7efc340f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:51.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 --2- 172.21.15.76:0/4165371279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f9160 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7efc28009fb0 tx=0x7efc28009b30).stop 2023-12-26T21:57:51.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 -- 172.21.15.76:0/4165371279 shutdown_connections 2023-12-26T21:57:51.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 --2- 172.21.15.76:0/4165371279 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f9160 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:51.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 -- 172.21.15.76:0/4165371279 >> 172.21.15.76:0/4165371279 conn(0x7efc340f0be0 msgr2=0x7efc340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:51.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 -- 172.21.15.76:0/4165371279 shutdown_connections 2023-12-26T21:57:51.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 -- 172.21.15.76:0/4165371279 wait complete. 2023-12-26T21:57:51.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 Processor -- start 2023-12-26T21:57:51.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.611+0000 7efc39758700 1 -- start start 2023-12-26T21:57:51.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc39758700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:51.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc39758700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc340f5060 con 0x7efc34060360 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc32ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50086/0 (socket says 172.21.15.76:50086) 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc32ffd700 1 -- 172.21.15.76:0/2809577244 learned_addr learned my addr 172.21.15.76:0/2809577244 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc32ffd700 1 -- 172.21.15.76:0/2809577244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc28005040 con 0x7efc34060360 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.612+0000 7efc32ffd700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f6be0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efc28009f80 tx=0x7efc2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.613+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc28009320 con 0x7efc34060360 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.613+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efc28013680 con 0x7efc34060360 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.613+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc2801eda0 con 0x7efc34060360 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.613+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc340f5200 con 0x7efc34060360 2023-12-26T21:57:51.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.613+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc340f5690 con 0x7efc34060360 2023-12-26T21:57:51.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.614+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efc2801e420 con 0x7efc34060360 2023-12-26T21:57:51.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.614+0000 7efc1b7fe700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc1c0310e0 0x7efc1c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:51.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.614+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efc280425c0 con 0x7efc34060360 2023-12-26T21:57:51.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.614+0000 7efc327fc700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc1c0310e0 0x7efc1c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:51.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.615+0000 7efc327fc700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc1c0310e0 0x7efc1c0335a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efc240096a0 tx=0x7efc24006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:51.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.616+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc340591e0 con 0x7efc34060360 2023-12-26T21:57:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.621+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efc2801e750 con 0x7efc34060360 2023-12-26T21:57:51.825 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.825+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc340f5cf0 con 0x7efc34060360 2023-12-26T21:57:51.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.826+0000 7efc1b7fe700 1 -- 172.21.15.76:0/2809577244 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc2800f340 con 0x7efc34060360 2023-12-26T21:57:51.826 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:51.826 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc1c0310e0 msgr2=0x7efc1c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc1c0310e0 0x7efc1c0335a0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7efc240096a0 tx=0x7efc24006b40).stop 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 msgr2=0x7efc340f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f6be0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7efc28009f80 tx=0x7efc2800b040).stop 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 shutdown_connections 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc1c0310e0 0x7efc1c0335a0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 --2- 172.21.15.76:0/2809577244 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc34060360 0x7efc340f6be0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 >> 172.21.15.76:0/2809577244 conn(0x7efc340f0be0 msgr2=0x7efc340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:51.828 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 shutdown_connections 2023-12-26T21:57:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:51.827+0000 7efc39758700 1 -- 172.21.15.76:0/2809577244 wait complete. 2023-12-26T21:57:51.829 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:53.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:53.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:51.332188+0000 mgr.smithi059.gwkiyc (mgr.14156) 35 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:51.827220+0000 mon.smithi059 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.76:0/2809577244' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:54.436 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 -- 172.21.15.76:0/2965035242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 msgr2=0x7f380c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 --2- 172.21.15.76:0/2965035242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c0f56f0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f3808009fb0 tx=0x7f3808009b30).stop 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 -- 172.21.15.76:0/2965035242 shutdown_connections 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 --2- 172.21.15.76:0/2965035242 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c0f56f0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 -- 172.21.15.76:0/2965035242 >> 172.21.15.76:0/2965035242 conn(0x7f380c0f0a30 msgr2=0x7f380c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 -- 172.21.15.76:0/2965035242 shutdown_connections 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.435+0000 7f38131cd700 1 -- 172.21.15.76:0/2965035242 wait complete. 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38131cd700 1 Processor -- start 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38131cd700 1 -- start start 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38131cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38131cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f380c106af0 con 0x7f380c0f52d0 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38121cb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38121cb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50102/0 (socket says 172.21.15.76:50102) 2023-12-26T21:57:54.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38121cb700 1 -- 172.21.15.76:0/3806551801 learned_addr learned my addr 172.21.15.76:0/3806551801 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38121cb700 1 -- 172.21.15.76:0/3806551801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3808005040 con 0x7f380c0f52d0 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.436+0000 7f38121cb700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c106600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f3808000c00 tx=0x7f380800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.437+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38080095f0 con 0x7f380c0f52d0 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.437+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f380c106cf0 con 0x7f380c0f52d0 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.437+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3808007e50 con 0x7f380c0f52d0 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.437+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3808014e70 con 0x7f380c0f52d0 2023-12-26T21:57:54.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.437+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f380c107180 con 0x7f380c0f52d0 2023-12-26T21:57:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.437+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3808025070 con 0x7f380c0f52d0 2023-12-26T21:57:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.438+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f380c045bb0 con 0x7f380c0f52d0 2023-12-26T21:57:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.438+0000 7f3802ffd700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37f8031090 0x7f37f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.438+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3808041d60 con 0x7f380c0f52d0 2023-12-26T21:57:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.438+0000 7f38119ca700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37f8031090 0x7f37f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:54.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.438+0000 7f38119ca700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37f8031090 0x7f37f8033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f37fc0096a0 tx=0x7f37fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:54.441 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.442+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f380800f4e0 con 0x7f380c0f52d0 2023-12-26T21:57:54.648 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.649+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f380c107f80 con 0x7f380c0f52d0 2023-12-26T21:57:54.649 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.649+0000 7f3802ffd700 1 -- 172.21.15.76:0/3806551801 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f380800f300 con 0x7f380c0f52d0 2023-12-26T21:57:54.649 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:54.649 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37f8031090 msgr2=0x7f37f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:54.650 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37f8031090 0x7f37f8033550 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f37fc0096a0 tx=0x7f37fc006b40).stop 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 msgr2=0x7f380c106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c106600 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f3808000c00 tx=0x7f380800b040).stop 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 shutdown_connections 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37f8031090 0x7f37f8033550 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 --2- 172.21.15.76:0/3806551801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f380c0f52d0 0x7f380c106600 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 >> 172.21.15.76:0/3806551801 conn(0x7f380c0f0a30 msgr2=0x7f380c05fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 shutdown_connections 2023-12-26T21:57:54.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:54.651+0000 7f38131cd700 1 -- 172.21.15.76:0/3806551801 wait complete. 2023-12-26T21:57:54.652 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:53.332566+0000 mgr.smithi059.gwkiyc (mgr.14156) 36 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:54.649800+0000 mon.smithi059 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.76:0/3806551801' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:56.047 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:56.048 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:55.332957+0000 mgr.smithi059.gwkiyc (mgr.14156) 37 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:57:57.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 -- 172.21.15.76:0/2842857740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 msgr2=0x7fe40c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 --2- 172.21.15.76:0/2842857740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c0f69b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fe400004760 tx=0x7fe400009ad0).stop 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 -- 172.21.15.76:0/2842857740 shutdown_connections 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 --2- 172.21.15.76:0/2842857740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c0f69b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 -- 172.21.15.76:0/2842857740 >> 172.21.15.76:0/2842857740 conn(0x7fe40c0f1cd0 msgr2=0x7fe40c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 -- 172.21.15.76:0/2842857740 shutdown_connections 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.306+0000 7fe411514700 1 -- 172.21.15.76:0/2842857740 wait complete. 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.307+0000 7fe411514700 1 Processor -- start 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.307+0000 7fe411514700 1 -- start start 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.307+0000 7fe411514700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.307+0000 7fe411514700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe40c069f00 con 0x7fe40c0f6590 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe40affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe40affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50730/0 (socket says 172.21.15.76:50730) 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe40affd700 1 -- 172.21.15.76:0/2508111584 learned_addr learned my addr 172.21.15.76:0/2508111584 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:57:57.308 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe40affd700 1 -- 172.21.15.76:0/2508111584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe400009d70 con 0x7fe40c0f6590 2023-12-26T21:57:57.309 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe40affd700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c06ba80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe400004170 tx=0x7fe400009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:57.309 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe40000e040 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe400008ef0 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe40c06a0a0 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe400014c90 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.308+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe40c06a530 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.309+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe40c0591e0 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.309+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe400014420 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.310+0000 7fe3f37fe700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe3f4031090 0x7fe3f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.310+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe4000421f0 con 0x7fe40c0f6590 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.310+0000 7fe40a7fc700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe3f4031090 0x7fe3f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:57:57.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.310+0000 7fe40a7fc700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe3f4031090 0x7fe3f4033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe3fc0096a0 tx=0x7fe3fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:57:57.314 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.315+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe400026300 con 0x7fe40c0f6590 2023-12-26T21:57:57.518 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.518+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe40c0fc080 con 0x7fe40c0f6590 2023-12-26T21:57:57.518 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.519+0000 7fe3f37fe700 1 -- 172.21.15.76:0/2508111584 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe400012b20 con 0x7fe40c0f6590 2023-12-26T21:57:57.518 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:57:57.519 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.520+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe3f4031090 msgr2=0x7fe3f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.520+0000 7fe411514700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe3f4031090 0x7fe3f4033550 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fe3fc0096a0 tx=0x7fe3fc006b40).stop 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.520+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 msgr2=0x7fe40c06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.520+0000 7fe411514700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c06ba80 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe400004170 tx=0x7fe400009ad0).stop 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.521+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 shutdown_connections 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.521+0000 7fe411514700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe3f4031090 0x7fe3f4033550 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:57.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.521+0000 7fe411514700 1 --2- 172.21.15.76:0/2508111584 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe40c0f6590 0x7fe40c06ba80 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:57:57.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.521+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 >> 172.21.15.76:0/2508111584 conn(0x7fe40c0f1cd0 msgr2=0x7fe40c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:57:57.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.521+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 shutdown_connections 2023-12-26T21:57:57.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:57:57.521+0000 7fe411514700 1 -- 172.21.15.76:0/2508111584 wait complete. 2023-12-26T21:57:57.521 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:57:58.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:57 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:57:57.519488+0000 mon.smithi059 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.76:0/2508111584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:57:58.929 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:57:58.929 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:57:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:57:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:57.333322+0000 mgr.smithi059.gwkiyc (mgr.14156) 38 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 -- 172.21.15.76:0/4031134330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 msgr2=0x7fcea00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 --2- 172.21.15.76:0/4031134330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea00f69b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fce90004760 tx=0x7fce90009ad0).stop 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 -- 172.21.15.76:0/4031134330 shutdown_connections 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 --2- 172.21.15.76:0/4031134330 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea00f69b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 -- 172.21.15.76:0/4031134330 >> 172.21.15.76:0/4031134330 conn(0x7fcea00f1cd0 msgr2=0x7fcea00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 -- 172.21.15.76:0/4031134330 shutdown_connections 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 -- 172.21.15.76:0/4031134330 wait complete. 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 Processor -- start 2023-12-26T21:58:00.146 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.146+0000 7fcea6233700 1 -- start start 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.147+0000 7fcea6233700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.147+0000 7fcea6233700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcea0106cb0 con 0x7fcea00f6590 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.147+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.147+0000 7fce9f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50750/0 (socket says 172.21.15.76:50750) 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.147+0000 7fce9f7fe700 1 -- 172.21.15.76:0/4121989326 learned_addr learned my addr 172.21.15.76:0/4121989326 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fce9f7fe700 1 -- 172.21.15.76:0/4121989326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce90009d70 con 0x7fcea00f6590 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fce9f7fe700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea01067c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fce90004170 tx=0x7fce90009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce9000e040 con 0x7fcea00f6590 2023-12-26T21:58:00.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fce90008ef0 con 0x7fcea00f6590 2023-12-26T21:58:00.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcea0106eb0 con 0x7fcea00f6590 2023-12-26T21:58:00.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fce90014c90 con 0x7fcea00f6590 2023-12-26T21:58:00.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.148+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcea0107340 con 0x7fcea00f6590 2023-12-26T21:58:00.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.149+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fce900283f0 con 0x7fcea00f6590 2023-12-26T21:58:00.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.149+0000 7fce9cff9700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fce8c031090 0x7fce8c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:00.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.149+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fce90042320 con 0x7fcea00f6590 2023-12-26T21:58:00.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.149+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcea0045bb0 con 0x7fcea00f6590 2023-12-26T21:58:00.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.149+0000 7fce96ffd700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fce8c031090 0x7fce8c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:00.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.150+0000 7fce96ffd700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fce8c031090 0x7fce8c033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fce880096a0 tx=0x7fce88006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:00.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.154+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fce900266c0 con 0x7fcea00f6590 2023-12-26T21:58:00.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.358+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcea0108060 con 0x7fcea00f6590 2023-12-26T21:58:00.358 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.359+0000 7fce9cff9700 1 -- 172.21.15.76:0/4121989326 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fce90026300 con 0x7fcea00f6590 2023-12-26T21:58:00.358 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:00.358 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fce8c031090 msgr2=0x7fce8c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fce8c031090 0x7fce8c033550 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fce880096a0 tx=0x7fce88006b40).stop 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 msgr2=0x7fcea01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea01067c0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fce90004170 tx=0x7fce90009ad0).stop 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 shutdown_connections 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fce8c031090 0x7fce8c033550 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 --2- 172.21.15.76:0/4121989326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcea00f6590 0x7fcea01067c0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 >> 172.21.15.76:0/4121989326 conn(0x7fcea00f1cd0 msgr2=0x7fcea00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.360+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 shutdown_connections 2023-12-26T21:58:00.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:00.361+0000 7fcea6233700 1 -- 172.21.15.76:0/4121989326 wait complete. 2023-12-26T21:58:00.361 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:57:59.333684+0000 mgr.smithi059.gwkiyc (mgr.14156) 39 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:00.359355+0000 mon.smithi059 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.76:0/4121989326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:01.764 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:01.764 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 -- 172.21.15.76:0/901654500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 msgr2=0x7fee180f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 --2- 172.21.15.76:0/901654500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee180f58a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7fee14009fb0 tx=0x7fee14009b30).stop 2023-12-26T21:58:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 -- 172.21.15.76:0/901654500 shutdown_connections 2023-12-26T21:58:02.992 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 --2- 172.21.15.76:0/901654500 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee180f58a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 -- 172.21.15.76:0/901654500 >> 172.21.15.76:0/901654500 conn(0x7fee180f0be0 msgr2=0x7fee180f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 -- 172.21.15.76:0/901654500 shutdown_connections 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 -- 172.21.15.76:0/901654500 wait complete. 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.992+0000 7fee20171700 1 Processor -- start 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee20171700 1 -- start start 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee20171700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee181094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee20171700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee18060320 con 0x7fee180f5480 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee1df0d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee181094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee1df0d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee181094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:50778/0 (socket says 172.21.15.76:50778) 2023-12-26T21:58:02.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee1df0d700 1 -- 172.21.15.76:0/3108417423 learned_addr learned my addr 172.21.15.76:0/3108417423 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:02.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee1df0d700 1 -- 172.21.15.76:0/3108417423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee14005040 con 0x7fee180f5480 2023-12-26T21:58:02.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee1df0d700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee181094f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fee140089f0 tx=0x7fee1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee14009320 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.993+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee18060520 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.994+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fee14013ec0 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.994+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee14014d30 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.994+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee18109ba0 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.994+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fee140095f0 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.994+0000 7fee0e7fc700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fee04031130 0x7fee040335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.994+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee18045bb0 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.995+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fee14042450 con 0x7fee180f5480 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.995+0000 7fee1d70c700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fee04031130 0x7fee040335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:02.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:02.995+0000 7fee1d70c700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fee04031130 0x7fee040335f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fee080096a0 tx=0x7fee08006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:02.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.000+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fee1401dd70 con 0x7fee180f5480 2023-12-26T21:58:03.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.204+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee1810a390 con 0x7fee180f5480 2023-12-26T21:58:03.204 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.205+0000 7fee0e7fc700 1 -- 172.21.15.76:0/3108417423 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fee1401d380 con 0x7fee180f5480 2023-12-26T21:58:03.204 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:03.204 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fee04031130 msgr2=0x7fee040335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fee04031130 0x7fee040335f0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fee080096a0 tx=0x7fee08006b40).stop 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 msgr2=0x7fee181094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee181094f0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fee140089f0 tx=0x7fee1400b040).stop 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 shutdown_connections 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fee04031130 0x7fee040335f0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 --2- 172.21.15.76:0/3108417423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fee180f5480 0x7fee181094f0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 >> 172.21.15.76:0/3108417423 conn(0x7fee180f0be0 msgr2=0x7fee180f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 shutdown_connections 2023-12-26T21:58:03.206 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:03.206+0000 7fee20171700 1 -- 172.21.15.76:0/3108417423 wait complete. 2023-12-26T21:58:03.207 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:01.334032+0000 mgr.smithi059.gwkiyc (mgr.14156) 40 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:04.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:03 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:03.205280+0000 mon.smithi059 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.76:0/3108417423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:04.618 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:04.619 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:03.334284+0000 mgr.smithi059.gwkiyc (mgr.14156) 41 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:05.858 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 -- 172.21.15.76:0/3462690007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 msgr2=0x7f14440f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:05.858 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 --2- 172.21.15.76:0/3462690007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f14440f58a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f1438009fb0 tx=0x7f1438009b30).stop 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 -- 172.21.15.76:0/3462690007 shutdown_connections 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 --2- 172.21.15.76:0/3462690007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f14440f58a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 -- 172.21.15.76:0/3462690007 >> 172.21.15.76:0/3462690007 conn(0x7f14440f0be0 msgr2=0x7f14440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 -- 172.21.15.76:0/3462690007 shutdown_connections 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 -- 172.21.15.76:0/3462690007 wait complete. 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 Processor -- start 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.858+0000 7f144abe4700 1 -- start start 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.859+0000 7f144abe4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f1444109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.859+0000 7f144abe4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1444060300 con 0x7f14440f5480 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.859+0000 7f1448980700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f1444109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.859+0000 7f1448980700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f1444109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46272/0 (socket says 172.21.15.76:46272) 2023-12-26T21:58:05.859 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.859+0000 7f1448980700 1 -- 172.21.15.76:0/2435119542 learned_addr learned my addr 172.21.15.76:0/2435119542 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:05.860 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.859+0000 7f1448980700 1 -- 172.21.15.76:0/2435119542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1438005040 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.860+0000 7f1448980700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f1444109510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1438000c00 tx=0x7f143800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.860+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f14380095f0 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.860+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1444060500 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.860+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1438007e50 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.860+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1438014e70 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.860+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1444109bc0 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.861+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1438025070 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.861+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14440591e0 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.861+0000 7f14417fa700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1434031090 0x7f1434033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.861+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1438041e00 con 0x7f14440f5480 2023-12-26T21:58:05.861 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.861+0000 7f1443fff700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1434031090 0x7f1434033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:05.862 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.861+0000 7f1443fff700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1434031090 0x7f1434033550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f14300096a0 tx=0x7f1430006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:05.865 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:05.866+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f143800f300 con 0x7f14440f5480 2023-12-26T21:58:06.072 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.073+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1444045bb0 con 0x7f14440f5480 2023-12-26T21:58:06.073 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.073+0000 7f14417fa700 1 -- 172.21.15.76:0/2435119542 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f143800f300 con 0x7f14440f5480 2023-12-26T21:58:06.073 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:06.073 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:06.074 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1434031090 msgr2=0x7f1434033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:06.074 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1434031090 0x7f1434033550 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f14300096a0 tx=0x7f1430006b40).stop 2023-12-26T21:58:06.074 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 msgr2=0x7f1444109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:06.074 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f1444109510 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f1438000c00 tx=0x7f143800b040).stop 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 shutdown_connections 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1434031090 0x7f1434033550 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 --2- 172.21.15.76:0/2435119542 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14440f5480 0x7f1444109510 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 >> 172.21.15.76:0/2435119542 conn(0x7f14440f0be0 msgr2=0x7f14440f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 shutdown_connections 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:06.075+0000 7f144abe4700 1 -- 172.21.15.76:0/2435119542 wait complete. 2023-12-26T21:58:06.075 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:05.334614+0000 mgr.smithi059.gwkiyc (mgr.14156) 42 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:06.074019+0000 mon.smithi059 (mon.0) 129 : audit [DBG] from='client.? 172.21.15.76:0/2435119542' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:07.471 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:07.471 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- 172.21.15.76:0/1184350197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 msgr2=0x7f61740f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 --2- 172.21.15.76:0/1184350197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f61740f58a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6164009fb0 tx=0x7f6164009b30).stop 2023-12-26T21:58:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- 172.21.15.76:0/1184350197 shutdown_connections 2023-12-26T21:58:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 --2- 172.21.15.76:0/1184350197 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f61740f58a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- 172.21.15.76:0/1184350197 >> 172.21.15.76:0/1184350197 conn(0x7f61740f0be0 msgr2=0x7f61740f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- 172.21.15.76:0/1184350197 shutdown_connections 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- 172.21.15.76:0/1184350197 wait complete. 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 Processor -- start 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- start start 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f6174109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.681+0000 7f6178ac6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6174060300 con 0x7f61740f5480 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f617259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f6174109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f617259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f6174109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46300/0 (socket says 172.21.15.76:46300) 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f617259c700 1 -- 172.21.15.76:0/2948638688 learned_addr learned my addr 172.21.15.76:0/2948638688 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f617259c700 1 -- 172.21.15.76:0/2948638688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6164005040 con 0x7f61740f5480 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f617259c700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f6174109510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f61640089f0 tx=0x7f616400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6164009320 con 0x7f61740f5480 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6174060500 con 0x7f61740f5480 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6164013ec0 con 0x7f61740f5480 2023-12-26T21:58:08.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6164014d30 con 0x7f61740f5480 2023-12-26T21:58:08.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.682+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6174109bc0 con 0x7f61740f5480 2023-12-26T21:58:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.683+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f61640095f0 con 0x7f61740f5480 2023-12-26T21:58:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.683+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61740591e0 con 0x7f61740f5480 2023-12-26T21:58:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.683+0000 7f616affd700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6160031090 0x7f6160033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.683+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f61640423e0 con 0x7f61740f5480 2023-12-26T21:58:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.683+0000 7f6171d9b700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6160031090 0x7f6160033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.684+0000 7f6171d9b700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6160031090 0x7f6160033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f615c0096a0 tx=0x7f615c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:08.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.688+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f616401d7f0 con 0x7f61740f5480 2023-12-26T21:58:08.893 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.893+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6174045bb0 con 0x7f61740f5480 2023-12-26T21:58:08.894 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.894+0000 7f616affd700 1 -- 172.21.15.76:0/2948638688 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f616401d7f0 con 0x7f61740f5480 2023-12-26T21:58:08.895 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:08.895 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6160031090 msgr2=0x7f6160033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6160031090 0x7f6160033550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f615c0096a0 tx=0x7f615c006b40).stop 2023-12-26T21:58:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 msgr2=0x7f6174109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f6174109510 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f61640089f0 tx=0x7f616400b040).stop 2023-12-26T21:58:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 shutdown_connections 2023-12-26T21:58:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6160031090 0x7f6160033550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 --2- 172.21.15.76:0/2948638688 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61740f5480 0x7f6174109510 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.895+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 >> 172.21.15.76:0/2948638688 conn(0x7f61740f0be0 msgr2=0x7f61740f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.896+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 shutdown_connections 2023-12-26T21:58:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:08.896+0000 7f6178ac6700 1 -- 172.21.15.76:0/2948638688 wait complete. 2023-12-26T21:58:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:07.335015+0000 mgr.smithi059.gwkiyc (mgr.14156) 43 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:10.281 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:10.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:10.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:09 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:08.895385+0000 mon.smithi059 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.76:0/2948638688' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:09.335283+0000 mgr.smithi059.gwkiyc (mgr.14156) 44 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:11.431 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 -- 172.21.15.76:0/4018623335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 msgr2=0x7f666c0f8ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:11.431 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 --2- 172.21.15.76:0/4018623335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c0f8ff0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f6660009fb0 tx=0x7f6660009b30).stop 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 -- 172.21.15.76:0/4018623335 shutdown_connections 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 --2- 172.21.15.76:0/4018623335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c0f8ff0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 -- 172.21.15.76:0/4018623335 >> 172.21.15.76:0/4018623335 conn(0x7f666c0f0a30 msgr2=0x7f666c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 -- 172.21.15.76:0/4018623335 shutdown_connections 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 -- 172.21.15.76:0/4018623335 wait complete. 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 Processor -- start 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.430+0000 7f6671972700 1 -- start start 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6671972700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c106610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6671972700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f666c106b00 con 0x7f666c060320 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6670970700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c106610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:11.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6670970700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c106610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46320/0 (socket says 172.21.15.76:46320) 2023-12-26T21:58:11.433 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6670970700 1 -- 172.21.15.76:0/1050278665 learned_addr learned my addr 172.21.15.76:0/1050278665 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:11.433 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6670970700 1 -- 172.21.15.76:0/1050278665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6660005040 con 0x7f666c060320 2023-12-26T21:58:11.434 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f6670970700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c106610 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6660009f80 tx=0x7f666000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.431+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6660009320 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6660013680 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f666c106d00 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f666001eda0 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f666c107190 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f666001e420 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f666c045bb0 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.432+0000 7f66697fa700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f665c0310e0 0x7f665c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.433+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6660042320 con 0x7f666c060320 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.433+0000 7f666bfff700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f665c0310e0 0x7f665c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:11.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.433+0000 7f666bfff700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f665c0310e0 0x7f665c0335a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f66580096a0 tx=0x7f6658006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:11.438 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.438+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6660014cc0 con 0x7f666c060320 2023-12-26T21:58:11.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.650+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f666c108040 con 0x7f666c060320 2023-12-26T21:58:11.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.651+0000 7f66697fa700 1 -- 172.21.15.76:0/1050278665 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6660004490 con 0x7f666c060320 2023-12-26T21:58:11.652 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:11.652 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.652+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f665c0310e0 msgr2=0x7f665c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.652+0000 7f6671972700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f665c0310e0 0x7f665c0335a0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f66580096a0 tx=0x7f6658006b40).stop 2023-12-26T21:58:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.652+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 msgr2=0x7f666c106610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.652+0000 7f6671972700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c106610 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f6660009f80 tx=0x7f666000b040).stop 2023-12-26T21:58:11.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.653+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 shutdown_connections 2023-12-26T21:58:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.653+0000 7f6671972700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f665c0310e0 0x7f665c0335a0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.653+0000 7f6671972700 1 --2- 172.21.15.76:0/1050278665 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f666c060320 0x7f666c106610 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.653+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 >> 172.21.15.76:0/1050278665 conn(0x7f666c0f0a30 msgr2=0x7f666c0f1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.653+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 shutdown_connections 2023-12-26T21:58:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:11.653+0000 7f6671972700 1 -- 172.21.15.76:0/1050278665 wait complete. 2023-12-26T21:58:11.654 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:12.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:11 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:11.652617+0000 mon.smithi059 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.76:0/1050278665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:13.059 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:13.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:13.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:11.335609+0000 mgr.smithi059.gwkiyc (mgr.14156) 45 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:14.251 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 -- 172.21.15.76:0/808661712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 msgr2=0x7f4df40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 --2- 172.21.15.76:0/808661712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df40f58a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f4de8009fb0 tx=0x7f4de8009b30).stop 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 -- 172.21.15.76:0/808661712 shutdown_connections 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 --2- 172.21.15.76:0/808661712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df40f58a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 -- 172.21.15.76:0/808661712 >> 172.21.15.76:0/808661712 conn(0x7f4df40f0be0 msgr2=0x7f4df40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 -- 172.21.15.76:0/808661712 shutdown_connections 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.249+0000 7f4dfabe0700 1 -- 172.21.15.76:0/808661712 wait complete. 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.250+0000 7f4dfabe0700 1 Processor -- start 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.250+0000 7f4dfabe0700 1 -- start start 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.250+0000 7f4dfabe0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df4109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.250+0000 7f4dfabe0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4df4060300 con 0x7f4df40f5480 2023-12-26T21:58:14.252 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df897c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df4109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df897c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df4109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46336/0 (socket says 172.21.15.76:46336) 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df897c700 1 -- 172.21.15.76:0/2000029967 learned_addr learned my addr 172.21.15.76:0/2000029967 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df897c700 1 -- 172.21.15.76:0/2000029967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4de8005040 con 0x7f4df40f5480 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df897c700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df4109510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f4de80089f0 tx=0x7f4de800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de8009320 con 0x7f4df40f5480 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4df4060500 con 0x7f4df40f5480 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4de8013ec0 con 0x7f4df40f5480 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4de8014d30 con 0x7f4df40f5480 2023-12-26T21:58:14.253 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.251+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4df4109bc0 con 0x7f4df40f5480 2023-12-26T21:58:14.254 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.252+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4de80095f0 con 0x7f4df40f5480 2023-12-26T21:58:14.254 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.252+0000 7f4df17fa700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4de4031090 0x7f4de4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:14.254 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.252+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4df40591e0 con 0x7f4df40f5480 2023-12-26T21:58:14.254 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.252+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4de80423e0 con 0x7f4df40f5480 2023-12-26T21:58:14.254 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.252+0000 7f4df3fff700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4de4031090 0x7f4de4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:14.254 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.253+0000 7f4df3fff700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4de4031090 0x7f4de4033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4de00096a0 tx=0x7f4de0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:14.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.257+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4de801d7f0 con 0x7f4df40f5480 2023-12-26T21:58:14.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.465+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4df4045bb0 con 0x7f4df40f5480 2023-12-26T21:58:14.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.466+0000 7f4df17fa700 1 -- 172.21.15.76:0/2000029967 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4de801d7f0 con 0x7f4df40f5480 2023-12-26T21:58:14.466 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:14.466 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4de4031090 msgr2=0x7f4de4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4de4031090 0x7f4de4033550 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f4de00096a0 tx=0x7f4de0006b40).stop 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 msgr2=0x7f4df4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df4109510 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f4de80089f0 tx=0x7f4de800b040).stop 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 shutdown_connections 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4de4031090 0x7f4de4033550 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 --2- 172.21.15.76:0/2000029967 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4df40f5480 0x7f4df4109510 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 >> 172.21.15.76:0/2000029967 conn(0x7f4df40f0be0 msgr2=0x7f4df40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 shutdown_connections 2023-12-26T21:58:14.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:14.467+0000 7f4dfabe0700 1 -- 172.21.15.76:0/2000029967 wait complete. 2023-12-26T21:58:14.469 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:13.335943+0000 mgr.smithi059.gwkiyc (mgr.14156) 46 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:14.467194+0000 mon.smithi059 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.76:0/2000029967' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:15.852 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:15.852 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 -- 172.21.15.76:0/1077896726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 msgr2=0x7f8e500f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 --2- 172.21.15.76:0/1077896726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e500f69b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f8e40004760 tx=0x7f8e40009ad0).stop 2023-12-26T21:58:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 -- 172.21.15.76:0/1077896726 shutdown_connections 2023-12-26T21:58:17.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 --2- 172.21.15.76:0/1077896726 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e500f69b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 -- 172.21.15.76:0/1077896726 >> 172.21.15.76:0/1077896726 conn(0x7f8e500f1cd0 msgr2=0x7f8e500f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 -- 172.21.15.76:0/1077896726 shutdown_connections 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 -- 172.21.15.76:0/1077896726 wait complete. 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.049+0000 7f8e565e7700 1 Processor -- start 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e565e7700 1 -- start start 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e565e7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e501067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e565e7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e50106cb0 con 0x7f8e500f6590 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e4ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e501067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e4ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e501067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48490/0 (socket says 172.21.15.76:48490) 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e4ffff700 1 -- 172.21.15.76:0/400712144 learned_addr learned my addr 172.21.15.76:0/400712144 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e4ffff700 1 -- 172.21.15.76:0/400712144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e40009d70 con 0x7f8e500f6590 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.050+0000 7f8e4ffff700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e501067c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8e40004170 tx=0x7f8e40009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:17.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.051+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e4000e040 con 0x7f8e500f6590 2023-12-26T21:58:17.053 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.051+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e50106eb0 con 0x7f8e500f6590 2023-12-26T21:58:17.053 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.051+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8e40008ef0 con 0x7f8e500f6590 2023-12-26T21:58:17.053 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.051+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e40014c90 con 0x7f8e500f6590 2023-12-26T21:58:17.053 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.051+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e50107340 con 0x7f8e500f6590 2023-12-26T21:58:17.054 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.051+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8e400283f0 con 0x7f8e500f6590 2023-12-26T21:58:17.054 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.052+0000 7f8e4cff9700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8e38031090 0x7f8e38033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:17.054 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.052+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8e400422b0 con 0x7f8e500f6590 2023-12-26T21:58:17.054 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.052+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e500591e0 con 0x7f8e500f6590 2023-12-26T21:58:17.054 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.052+0000 7f8e4f7fe700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8e38031090 0x7f8e38033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:17.054 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.052+0000 7f8e4f7fe700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8e38031090 0x7f8e38033550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8e440096a0 tx=0x7f8e44006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:17.057 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.057+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8e400268f0 con 0x7f8e500f6590 2023-12-26T21:58:17.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.265+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8e50045bb0 con 0x7f8e500f6590 2023-12-26T21:58:17.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.266+0000 7f8e4cff9700 1 -- 172.21.15.76:0/400712144 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8e40026300 con 0x7f8e500f6590 2023-12-26T21:58:17.266 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:17.267 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:17.267 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8e38031090 msgr2=0x7f8e38033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8e38031090 0x7f8e38033550 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f8e440096a0 tx=0x7f8e44006b40).stop 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 msgr2=0x7f8e501067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e501067c0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f8e40004170 tx=0x7f8e40009ad0).stop 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 shutdown_connections 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8e38031090 0x7f8e38033550 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 --2- 172.21.15.76:0/400712144 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8e500f6590 0x7f8e501067c0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 >> 172.21.15.76:0/400712144 conn(0x7f8e500f1cd0 msgr2=0x7f8e500f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 shutdown_connections 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:17.267+0000 7f8e565e7700 1 -- 172.21.15.76:0/400712144 wait complete. 2023-12-26T21:58:17.268 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:17.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:15.336336+0000 mgr.smithi059.gwkiyc (mgr.14156) 47 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:18.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:17 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:17.267226+0000 mon.smithi059 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.76:0/400712144' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:18.667 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:18.668 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:19.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:17.336671+0000 mgr.smithi059.gwkiyc (mgr.14156) 48 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 -- 172.21.15.76:0/632815138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 msgr2=0x7ff7c00f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 --2- 172.21.15.76:0/632815138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c00f67f0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff7b4009fb0 tx=0x7ff7b4009b30).stop 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 -- 172.21.15.76:0/632815138 shutdown_connections 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 --2- 172.21.15.76:0/632815138 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c00f67f0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 -- 172.21.15.76:0/632815138 >> 172.21.15.76:0/632815138 conn(0x7ff7c00f1b10 msgr2=0x7ff7c00f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 -- 172.21.15.76:0/632815138 shutdown_connections 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 -- 172.21.15.76:0/632815138 wait complete. 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 Processor -- start 2023-12-26T21:58:19.934 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.933+0000 7ff7c5c81700 1 -- start start 2023-12-26T21:58:19.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c5c81700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c0106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:19.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c5c81700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff7c0106b30 con 0x7ff7c00f63d0 2023-12-26T21:58:19.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c4c7f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c0106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c4c7f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c0106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48500/0 (socket says 172.21.15.76:48500) 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c4c7f700 1 -- 172.21.15.76:0/3039795807 learned_addr learned my addr 172.21.15.76:0/3039795807 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c4c7f700 1 -- 172.21.15.76:0/3039795807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff7b4005040 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.934+0000 7ff7c4c7f700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c0106640 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff7b40089f0 tx=0x7ff7b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.935+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7b4009320 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.935+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff7c0106d30 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.935+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff7b4013ec0 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.935+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7b4014d30 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.935+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff7c01071c0 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.935+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff7b40095f0 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.936+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff7c0045bb0 con 0x7ff7c00f63d0 2023-12-26T21:58:19.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.936+0000 7ff7bd7fa700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff7b0031090 0x7ff7b0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.936+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff7b40424a0 con 0x7ff7c00f63d0 2023-12-26T21:58:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.936+0000 7ff7bffff700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff7b0031090 0x7ff7b0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:19.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.936+0000 7ff7bffff700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff7b0031090 0x7ff7b0033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff7ac0096a0 tx=0x7ff7ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:19.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:19.940+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff7b401d7f0 con 0x7ff7c00f63d0 2023-12-26T21:58:20.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.147+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff7c0108070 con 0x7ff7c00f63d0 2023-12-26T21:58:20.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.147+0000 7ff7bd7fa700 1 -- 172.21.15.76:0/3039795807 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff7b401d7f0 con 0x7ff7c00f63d0 2023-12-26T21:58:20.148 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:20.148 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff7b0031090 msgr2=0x7ff7b0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff7b0031090 0x7ff7b0033550 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7ff7ac0096a0 tx=0x7ff7ac006b40).stop 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 msgr2=0x7ff7c0106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c0106640 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7ff7b40089f0 tx=0x7ff7b400b040).stop 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 shutdown_connections 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff7b0031090 0x7ff7b0033550 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 --2- 172.21.15.76:0/3039795807 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff7c00f63d0 0x7ff7c0106640 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 >> 172.21.15.76:0/3039795807 conn(0x7ff7c00f1b10 msgr2=0x7ff7c00f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 shutdown_connections 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:20.149+0000 7ff7c5c81700 1 -- 172.21.15.76:0/3039795807 wait complete. 2023-12-26T21:58:20.150 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:19.336944+0000 mgr.smithi059.gwkiyc (mgr.14156) 49 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:20.148860+0000 mon.smithi059 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.76:0/3039795807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:21.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:21.540 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:22.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 -- 172.21.15.76:0/2505115969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 msgr2=0x7fa85c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 --2- 172.21.15.76:0/2505115969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c0f58a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fa858009fb0 tx=0x7fa858009b30).stop 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 -- 172.21.15.76:0/2505115969 shutdown_connections 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 --2- 172.21.15.76:0/2505115969 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c0f58a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 -- 172.21.15.76:0/2505115969 >> 172.21.15.76:0/2505115969 conn(0x7fa85c0f0be0 msgr2=0x7fa85c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 -- 172.21.15.76:0/2505115969 shutdown_connections 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.714+0000 7fa864711700 1 -- 172.21.15.76:0/2505115969 wait complete. 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa864711700 1 Processor -- start 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa864711700 1 -- start start 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa864711700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c1089c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa864711700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c108eb0 con 0x7fa85c0f5480 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa8624ad700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c1089c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa8624ad700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c1089c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48504/0 (socket says 172.21.15.76:48504) 2023-12-26T21:58:22.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.715+0000 7fa8624ad700 1 -- 172.21.15.76:0/1819617212 learned_addr learned my addr 172.21.15.76:0/1819617212 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa8624ad700 1 -- 172.21.15.76:0/1819617212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa858005040 con 0x7fa85c0f5480 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa8624ad700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c1089c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa8580089f0 tx=0x7fa85800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa858009320 con 0x7fa85c0f5480 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c1090b0 con 0x7fa85c0f5480 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa858013ec0 con 0x7fa85c0f5480 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa858014d30 con 0x7fa85c0f5480 2023-12-26T21:58:22.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.716+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c109540 con 0x7fa85c0f5480 2023-12-26T21:58:22.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.717+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa8580095f0 con 0x7fa85c0f5480 2023-12-26T21:58:22.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.717+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c045bb0 con 0x7fa85c0f5480 2023-12-26T21:58:22.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.717+0000 7fa852ffd700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa848031070 0x7fa848033530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:22.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.717+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa8580423e0 con 0x7fa85c0f5480 2023-12-26T21:58:22.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.717+0000 7fa861cac700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa848031070 0x7fa848033530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:22.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.718+0000 7fa861cac700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa848031070 0x7fa848033530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:22.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.722+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa85801d7f0 con 0x7fa85c0f5480 2023-12-26T21:58:22.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.932+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa85c10a3f0 con 0x7fa85c0f5480 2023-12-26T21:58:22.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.933+0000 7fa852ffd700 1 -- 172.21.15.76:0/1819617212 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa85801d7f0 con 0x7fa85c0f5480 2023-12-26T21:58:22.934 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:22.934 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.934+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa848031070 msgr2=0x7fa848033530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.934+0000 7fa864711700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa848031070 0x7fa848033530 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).stop 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.934+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 msgr2=0x7fa85c1089c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.934+0000 7fa864711700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c1089c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fa8580089f0 tx=0x7fa85800b040).stop 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.935+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 shutdown_connections 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.935+0000 7fa864711700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa848031070 0x7fa848033530 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.935+0000 7fa864711700 1 --2- 172.21.15.76:0/1819617212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa85c0f5480 0x7fa85c1089c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:22.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.935+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 >> 172.21.15.76:0/1819617212 conn(0x7fa85c0f0be0 msgr2=0x7fa85c0f1740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.935+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 shutdown_connections 2023-12-26T21:58:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:22.935+0000 7fa864711700 1 -- 172.21.15.76:0/1819617212 wait complete. 2023-12-26T21:58:22.936 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:21.337308+0000 mgr.smithi059.gwkiyc (mgr.14156) 50 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:24.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:22.934454+0000 mon.smithi059 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.76:0/1819617212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:24.345 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:24.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:23.337622+0000 mgr.smithi059.gwkiyc (mgr.14156) 51 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:25.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/3933336952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 msgr2=0x7fb3c80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:25.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 --2- 172.21.15.76:0/3933336952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c80f58a0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7fb3b8009fb0 tx=0x7fb3b8009b30).stop 2023-12-26T21:58:25.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/3933336952 shutdown_connections 2023-12-26T21:58:25.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 --2- 172.21.15.76:0/3933336952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c80f58a0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:25.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/3933336952 >> 172.21.15.76:0/3933336952 conn(0x7fb3c80f0be0 msgr2=0x7fb3c80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:25.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/3933336952 shutdown_connections 2023-12-26T21:58:25.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.563+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/3933336952 wait complete. 2023-12-26T21:58:25.564 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.564+0000 7fb3ce0dc700 1 Processor -- start 2023-12-26T21:58:25.564 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.564+0000 7fb3ce0dc700 1 -- start start 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.564+0000 7fb3ce0dc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c81094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.564+0000 7fb3ce0dc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8060320 con 0x7fb3c80f5480 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.564+0000 7fb3c77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c81094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c77fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c81094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52552/0 (socket says 172.21.15.76:52552) 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c77fe700 1 -- 172.21.15.76:0/1301227786 learned_addr learned my addr 172.21.15.76:0/1301227786 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c77fe700 1 -- 172.21.15.76:0/1301227786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b8005040 con 0x7fb3c80f5480 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c77fe700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c81094f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb3b80089f0 tx=0x7fb3b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3b8009320 con 0x7fb3c80f5480 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c8060520 con 0x7fb3c80f5480 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb3b8013ec0 con 0x7fb3c80f5480 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3b8014d30 con 0x7fb3c80f5480 2023-12-26T21:58:25.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.565+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c8109ba0 con 0x7fb3c80f5480 2023-12-26T21:58:25.566 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.566+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb3b80095f0 con 0x7fb3c80f5480 2023-12-26T21:58:25.566 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.566+0000 7fb3c4ff9700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb3b4031090 0x7fb3b4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:25.566 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.566+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb3b80424e0 con 0x7fb3c80f5480 2023-12-26T21:58:25.566 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.566+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8045bb0 con 0x7fb3c80f5480 2023-12-26T21:58:25.566 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.566+0000 7fb3beffd700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb3b4031090 0x7fb3b4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:25.567 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.567+0000 7fb3beffd700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb3b4031090 0x7fb3b4033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:25.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.571+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb3b801d7f0 con 0x7fb3c80f5480 2023-12-26T21:58:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.777+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3c80fb060 con 0x7fb3c80f5480 2023-12-26T21:58:25.777 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.778+0000 7fb3c4ff9700 1 -- 172.21.15.76:0/1301227786 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb3b802b020 con 0x7fb3c80f5480 2023-12-26T21:58:25.777 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:25.778 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb3b4031090 msgr2=0x7fb3b4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb3b4031090 0x7fb3b4033550 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fb3b00096a0 tx=0x7fb3b0006b40).stop 2023-12-26T21:58:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 msgr2=0x7fb3c81094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c81094f0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7fb3b80089f0 tx=0x7fb3b800b040).stop 2023-12-26T21:58:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 shutdown_connections 2023-12-26T21:58:25.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb3b4031090 0x7fb3b4033550 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:25.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 --2- 172.21.15.76:0/1301227786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb3c80f5480 0x7fb3c81094f0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:25.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 >> 172.21.15.76:0/1301227786 conn(0x7fb3c80f0be0 msgr2=0x7fb3c80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:25.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 shutdown_connections 2023-12-26T21:58:25.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:25.780+0000 7fb3ce0dc700 1 -- 172.21.15.76:0/1301227786 wait complete. 2023-12-26T21:58:25.780 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:26.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:25.778661+0000 mon.smithi059 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.76:0/1301227786' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:27.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:27.182 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:27.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:25.337963+0000 mgr.smithi059.gwkiyc (mgr.14156) 52 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:28.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 -- 172.21.15.76:0/1464462110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 msgr2=0x7ff0d80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 --2- 172.21.15.76:0/1464462110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d80f69b0 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7ff0c8009fb0 tx=0x7ff0c8009b30).stop 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 -- 172.21.15.76:0/1464462110 shutdown_connections 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 --2- 172.21.15.76:0/1464462110 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d80f69b0 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 -- 172.21.15.76:0/1464462110 >> 172.21.15.76:0/1464462110 conn(0x7ff0d80f1cf0 msgr2=0x7ff0d80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 -- 172.21.15.76:0/1464462110 shutdown_connections 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.415+0000 7ff0de576700 1 -- 172.21.15.76:0/1464462110 wait complete. 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.416+0000 7ff0de576700 1 Processor -- start 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.416+0000 7ff0de576700 1 -- start start 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.416+0000 7ff0de576700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d81067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:28.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.416+0000 7ff0de576700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d8106cc0 con 0x7ff0d80f6590 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.416+0000 7ff0d7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d81067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d81067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52558/0 (socket says 172.21.15.76:52558) 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d7fff700 1 -- 172.21.15.76:0/2056768685 learned_addr learned my addr 172.21.15.76:0/2056768685 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d7fff700 1 -- 172.21.15.76:0/2056768685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c8005040 con 0x7ff0d80f6590 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d7fff700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d81067d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff0c80089f0 tx=0x7ff0c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c8009320 con 0x7ff0d80f6590 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d8106ec0 con 0x7ff0d80f6590 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff0c8013ec0 con 0x7ff0d80f6590 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c8014d30 con 0x7ff0d80f6590 2023-12-26T21:58:28.418 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.417+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d8107350 con 0x7ff0d80f6590 2023-12-26T21:58:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.418+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff0c8014420 con 0x7ff0d80f6590 2023-12-26T21:58:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.418+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d8045bb0 con 0x7ff0d80f6590 2023-12-26T21:58:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.418+0000 7ff0d4ff9700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff0c4031090 0x7ff0c4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.418+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff0c8042400 con 0x7ff0d80f6590 2023-12-26T21:58:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.419+0000 7ff0d77fe700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff0c4031090 0x7ff0c4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:28.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.419+0000 7ff0d77fe700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff0c4031090 0x7ff0c4033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff0c00096a0 tx=0x7ff0c0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:28.422 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.423+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff0c80268c0 con 0x7ff0d80f6590 2023-12-26T21:58:28.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.630+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff0d8108200 con 0x7ff0d80f6590 2023-12-26T21:58:28.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.631+0000 7ff0d4ff9700 1 -- 172.21.15.76:0/2056768685 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff0c801d380 con 0x7ff0d80f6590 2023-12-26T21:58:28.631 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:28.631 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff0c4031090 msgr2=0x7ff0c4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff0c4031090 0x7ff0c4033550 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7ff0c00096a0 tx=0x7ff0c0006b40).stop 2023-12-26T21:58:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 msgr2=0x7ff0d81067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:28.632 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d81067d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7ff0c80089f0 tx=0x7ff0c800b040).stop 2023-12-26T21:58:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 shutdown_connections 2023-12-26T21:58:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff0c4031090 0x7ff0c4033550 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 --2- 172.21.15.76:0/2056768685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff0d80f6590 0x7ff0d81067d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.632+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 >> 172.21.15.76:0/2056768685 conn(0x7ff0d80f1cf0 msgr2=0x7ff0d80f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.633+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 shutdown_connections 2023-12-26T21:58:28.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:28.633+0000 7ff0de576700 1 -- 172.21.15.76:0/2056768685 wait complete. 2023-12-26T21:58:28.634 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:27.338402+0000 mgr.smithi059.gwkiyc (mgr.14156) 53 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:28.631394+0000 mon.smithi059 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.76:0/2056768685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:30.018 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:30.019 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:31.256 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 -- 172.21.15.76:0/718442917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 msgr2=0x7f85040f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 --2- 172.21.15.76:0/718442917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f85040f69b0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f84f8004760 tx=0x7f84f8009ad0).stop 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 -- 172.21.15.76:0/718442917 shutdown_connections 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 --2- 172.21.15.76:0/718442917 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f85040f69b0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 -- 172.21.15.76:0/718442917 >> 172.21.15.76:0/718442917 conn(0x7f85040f1cd0 msgr2=0x7f85040f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 -- 172.21.15.76:0/718442917 shutdown_connections 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.255+0000 7f8509a23700 1 -- 172.21.15.76:0/718442917 wait complete. 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8509a23700 1 Processor -- start 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8509a23700 1 -- start start 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8509a23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f8504108a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8509a23700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8504108f00 con 0x7f85040f6590 2023-12-26T21:58:31.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8502ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f8504108a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8502ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f8504108a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52576/0 (socket says 172.21.15.76:52576) 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.256+0000 7f8502ffd700 1 -- 172.21.15.76:0/2721535822 learned_addr learned my addr 172.21.15.76:0/2721535822 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f8502ffd700 1 -- 172.21.15.76:0/2721535822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f8009d70 con 0x7f85040f6590 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f8502ffd700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f8504108a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f84f8004170 tx=0x7f84f8009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f800e040 con 0x7f85040f6590 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8504109100 con 0x7f85040f6590 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f84f8008ef0 con 0x7f85040f6590 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84f8014c90 con 0x7f85040f6590 2023-12-26T21:58:31.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.257+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8504109590 con 0x7f85040f6590 2023-12-26T21:58:31.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.258+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f84f8014420 con 0x7f85040f6590 2023-12-26T21:58:31.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.258+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8504045bb0 con 0x7f85040f6590 2023-12-26T21:58:31.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.258+0000 7f84eb7fe700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f84ec031090 0x7f84ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:31.260 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.258+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f84f8042240 con 0x7f85040f6590 2023-12-26T21:58:31.260 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.258+0000 7f85027fc700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f84ec031090 0x7f84ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:31.260 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.259+0000 7f85027fc700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f84ec031090 0x7f84ec033550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f84f40096a0 tx=0x7f84f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:31.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.263+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f84f8026300 con 0x7f85040f6590 2023-12-26T21:58:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:29.338744+0000 mgr.smithi059.gwkiyc (mgr.14156) 54 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:31.469 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.470+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f850410a3f0 con 0x7f85040f6590 2023-12-26T21:58:31.470 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.470+0000 7f84eb7fe700 1 -- 172.21.15.76:0/2721535822 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f84f8012b20 con 0x7f85040f6590 2023-12-26T21:58:31.470 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:31.470 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:31.471 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f84ec031090 msgr2=0x7f84ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f84ec031090 0x7f84ec033550 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f84f40096a0 tx=0x7f84f4006b40).stop 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 msgr2=0x7f8504108a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f8504108a10 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f84f8004170 tx=0x7f84f8009ad0).stop 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 shutdown_connections 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f84ec031090 0x7f84ec033550 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 --2- 172.21.15.76:0/2721535822 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85040f6590 0x7f8504108a10 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 >> 172.21.15.76:0/2721535822 conn(0x7f85040f1cd0 msgr2=0x7f85040f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 shutdown_connections 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:31.472+0000 7f8509a23700 1 -- 172.21.15.76:0/2721535822 wait complete. 2023-12-26T21:58:31.472 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:32.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:31.470995+0000 mon.smithi059 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.76:0/2721535822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:32.860 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:32.860 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:31.339085+0000 mgr.smithi059.gwkiyc (mgr.14156) 55 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:34.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 -- 172.21.15.76:0/1149904078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 msgr2=0x7f26e40f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 --2- 172.21.15.76:0/1149904078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f9160 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f26d4009fb0 tx=0x7f26d4009b30).stop 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 -- 172.21.15.76:0/1149904078 shutdown_connections 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 --2- 172.21.15.76:0/1149904078 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f9160 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 -- 172.21.15.76:0/1149904078 >> 172.21.15.76:0/1149904078 conn(0x7f26e40f0be0 msgr2=0x7f26e40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 -- 172.21.15.76:0/1149904078 shutdown_connections 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.090+0000 7f26ea420700 1 -- 172.21.15.76:0/1149904078 wait complete. 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26ea420700 1 Processor -- start 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26ea420700 1 -- start start 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26ea420700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26ea420700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e40f5060 con 0x7f26e4060360 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:34.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26e3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52598/0 (socket says 172.21.15.76:52598) 2023-12-26T21:58:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26e3fff700 1 -- 172.21.15.76:0/3853701072 learned_addr learned my addr 172.21.15.76:0/3853701072 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:34.092 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.091+0000 7f26e3fff700 1 -- 172.21.15.76:0/3853701072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d4005040 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26e3fff700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f6be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f26d4000c00 tx=0x7f26d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d40095f0 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e40f5200 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f26d4007e50 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f26d4014e70 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e40f5690 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.092+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f26d4025070 con 0x7f26e4060360 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.093+0000 7f26e0ff9700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f26cc031090 0x7f26cc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:34.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.093+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f26d4041d60 con 0x7f26e4060360 2023-12-26T21:58:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.093+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e40591e0 con 0x7f26e4060360 2023-12-26T21:58:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.093+0000 7f26e37fe700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f26cc031090 0x7f26cc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:34.094 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.093+0000 7f26e37fe700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f26cc031090 0x7f26cc033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f26d80096a0 tx=0x7f26d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:34.097 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.098+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f26d400f4e0 con 0x7f26e4060360 2023-12-26T21:58:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.303+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f26e4045bb0 con 0x7f26e4060360 2023-12-26T21:58:34.303 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.304+0000 7f26e0ff9700 1 -- 172.21.15.76:0/3853701072 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f26d402d090 con 0x7f26e4060360 2023-12-26T21:58:34.304 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:34.304 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f26cc031090 msgr2=0x7f26cc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f26cc031090 0x7f26cc033550 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f26d80096a0 tx=0x7f26d8006b40).stop 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 msgr2=0x7f26e40f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f6be0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f26d4000c00 tx=0x7f26d400b040).stop 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 shutdown_connections 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f26cc031090 0x7f26cc033550 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 --2- 172.21.15.76:0/3853701072 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f26e4060360 0x7f26e40f6be0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 >> 172.21.15.76:0/3853701072 conn(0x7f26e40f0be0 msgr2=0x7f26e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 shutdown_connections 2023-12-26T21:58:34.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:34.306+0000 7f26ea420700 1 -- 172.21.15.76:0/3853701072 wait complete. 2023-12-26T21:58:34.307 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:33.339353+0000 mgr.smithi059.gwkiyc (mgr.14156) 56 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:34.304730+0000 mon.smithi059 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.76:0/3853701072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:35.640 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:35.640 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- 172.21.15.76:0/1573280313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 msgr2=0x7fa9740f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 --2- 172.21.15.76:0/1573280313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa9740f56f0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fa968009fb0 tx=0x7fa968009b30).stop 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- 172.21.15.76:0/1573280313 shutdown_connections 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 --2- 172.21.15.76:0/1573280313 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa9740f56f0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- 172.21.15.76:0/1573280313 >> 172.21.15.76:0/1573280313 conn(0x7fa9740f0a30 msgr2=0x7fa9740f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- 172.21.15.76:0/1573280313 shutdown_connections 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- 172.21.15.76:0/1573280313 wait complete. 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 Processor -- start 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- start start 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa974106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:36.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.882+0000 7fa97a04c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa974106b10 con 0x7fa9740f52d0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa97904a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa974106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa97904a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa974106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52216/0 (socket says 172.21.15.76:52216) 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa97904a700 1 -- 172.21.15.76:0/636207241 learned_addr learned my addr 172.21.15.76:0/636207241 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa97904a700 1 -- 172.21.15.76:0/636207241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa968005040 con 0x7fa9740f52d0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa97904a700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa974106620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa9680089f0 tx=0x7fa96800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa968009320 con 0x7fa9740f52d0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa974106d10 con 0x7fa9740f52d0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa968013ec0 con 0x7fa9740f52d0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.883+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa968014d30 con 0x7fa9740f52d0 2023-12-26T21:58:36.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.884+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa9741071a0 con 0x7fa9740f52d0 2023-12-26T21:58:36.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.884+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa968014420 con 0x7fa9740f52d0 2023-12-26T21:58:36.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.884+0000 7fa965ffb700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa960031090 0x7fa960033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:36.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.884+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa974045bb0 con 0x7fa9740f52d0 2023-12-26T21:58:36.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.885+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa9680424a0 con 0x7fa9740f52d0 2023-12-26T21:58:36.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.885+0000 7fa978849700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa960031090 0x7fa960033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:36.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.885+0000 7fa978849700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa960031090 0x7fa960033550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa9700096a0 tx=0x7fa970006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:36.889 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:36.890+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa96801d7f0 con 0x7fa9740f52d0 2023-12-26T21:58:37.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.097+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa974107fc0 con 0x7fa9740f52d0 2023-12-26T21:58:37.097 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.098+0000 7fa965ffb700 1 -- 172.21.15.76:0/636207241 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa96801d7f0 con 0x7fa9740f52d0 2023-12-26T21:58:37.097 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:37.097 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa960031090 msgr2=0x7fa960033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa960031090 0x7fa960033550 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fa9700096a0 tx=0x7fa970006b40).stop 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 msgr2=0x7fa974106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa974106620 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fa9680089f0 tx=0x7fa96800b040).stop 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 shutdown_connections 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa960031090 0x7fa960033550 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 --2- 172.21.15.76:0/636207241 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa9740f52d0 0x7fa974106620 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 >> 172.21.15.76:0/636207241 conn(0x7fa9740f0a30 msgr2=0x7fa97405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:37.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 shutdown_connections 2023-12-26T21:58:37.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:37.099+0000 7fa97a04c700 1 -- 172.21.15.76:0/636207241 wait complete. 2023-12-26T21:58:37.100 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:35.339639+0000 mgr.smithi059.gwkiyc (mgr.14156) 57 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:38.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:37 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:37.098277+0000 mon.smithi059 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.76:0/636207241' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:38.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:38.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:39.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:37.339953+0000 mgr.smithi059.gwkiyc (mgr.14156) 58 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 -- 172.21.15.76:0/1672744981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 msgr2=0x7f72140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 --2- 172.21.15.76:0/1672744981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72140f58a0 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f7208009fb0 tx=0x7f7208009b30).stop 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 -- 172.21.15.76:0/1672744981 shutdown_connections 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 --2- 172.21.15.76:0/1672744981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72140f58a0 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 -- 172.21.15.76:0/1672744981 >> 172.21.15.76:0/1672744981 conn(0x7f72140f0be0 msgr2=0x7f72140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 -- 172.21.15.76:0/1672744981 shutdown_connections 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 -- 172.21.15.76:0/1672744981 wait complete. 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.676+0000 7f72191bb700 1 Processor -- start 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f72191bb700 1 -- start start 2023-12-26T21:58:39.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f72191bb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72141094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:39.678 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f72191bb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7214060320 con 0x7f72140f5480 2023-12-26T21:58:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f7212d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72141094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f7212d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72141094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52238/0 (socket says 172.21.15.76:52238) 2023-12-26T21:58:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f7212d9d700 1 -- 172.21.15.76:0/1733919304 learned_addr learned my addr 172.21.15.76:0/1733919304 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f7212d9d700 1 -- 172.21.15.76:0/1733919304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7208005040 con 0x7f72140f5480 2023-12-26T21:58:39.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f7212d9d700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72141094f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7208000c00 tx=0x7f720800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72080095f0 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.677+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7208007e50 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.678+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7214060520 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.678+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7208014e70 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.678+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7214109ba0 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.678+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7208025070 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.678+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7214045bb0 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.678+0000 7f71fb7fe700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71fc031090 0x7f71fc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.679+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7208041dd0 con 0x7f72140f5480 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.679+0000 7f721259c700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71fc031090 0x7f71fc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:39.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.679+0000 7f721259c700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71fc031090 0x7f71fc033550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f72040096a0 tx=0x7f7204006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:39.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.683+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f720800f4e0 con 0x7f72140f5480 2023-12-26T21:58:39.890 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.889+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72140faf70 con 0x7f72140f5480 2023-12-26T21:58:39.890 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.890+0000 7f71fb7fe700 1 -- 172.21.15.76:0/1733919304 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f720800f300 con 0x7f72140f5480 2023-12-26T21:58:39.890 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:39.891 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:39.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71fc031090 msgr2=0x7f71fc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:39.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71fc031090 0x7f71fc033550 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f72040096a0 tx=0x7f7204006b40).stop 2023-12-26T21:58:39.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 msgr2=0x7f72141094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:39.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72141094f0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7208000c00 tx=0x7f720800b040).stop 2023-12-26T21:58:39.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 shutdown_connections 2023-12-26T21:58:39.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71fc031090 0x7f71fc033550 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:39.893 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 --2- 172.21.15.76:0/1733919304 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f72140f5480 0x7f72141094f0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:39.893 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 >> 172.21.15.76:0/1733919304 conn(0x7f72140f0be0 msgr2=0x7f72140f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:39.893 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 shutdown_connections 2023-12-26T21:58:39.893 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:39.892+0000 7f72191bb700 1 -- 172.21.15.76:0/1733919304 wait complete. 2023-12-26T21:58:39.893 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:41.280 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:41.281 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:39.340294+0000 mgr.smithi059.gwkiyc (mgr.14156) 59 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:39.891551+0000 mon.smithi059 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.76:0/1733919304' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:42.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/1763436545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 msgr2=0x7f4a080f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 --2- 172.21.15.76:0/1763436545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a080f58d0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f49fc009fb0 tx=0x7f49fc009b30).stop 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/1763436545 shutdown_connections 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 --2- 172.21.15.76:0/1763436545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a080f58d0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/1763436545 >> 172.21.15.76:0/1763436545 conn(0x7f4a080f0c10 msgr2=0x7f4a080f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/1763436545 shutdown_connections 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.517+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/1763436545 wait complete. 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a0ddeb700 1 Processor -- start 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a0ddeb700 1 -- start start 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a0ddeb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a081056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a0ddeb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a08105b90 con 0x7f4a080f54b0 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a077fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a081056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a077fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a081056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52252/0 (socket says 172.21.15.76:52252) 2023-12-26T21:58:42.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.518+0000 7f4a077fe700 1 -- 172.21.15.76:0/427710004 learned_addr learned my addr 172.21.15.76:0/427710004 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f4a077fe700 1 -- 172.21.15.76:0/427710004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49fc005040 con 0x7f4a080f54b0 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f4a077fe700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a081056a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f49fc0089f0 tx=0x7f49fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49fc009320 con 0x7f4a080f54b0 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a08105d90 con 0x7f4a080f54b0 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f49fc013ec0 con 0x7f4a080f54b0 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49fc014d30 con 0x7f4a080f54b0 2023-12-26T21:58:42.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.519+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a08106220 con 0x7f4a080f54b0 2023-12-26T21:58:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.520+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f49fc014420 con 0x7f4a080f54b0 2023-12-26T21:58:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.520+0000 7f49effff700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49f0031090 0x7f49f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.520+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a08045bb0 con 0x7f4a080f54b0 2023-12-26T21:58:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.520+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f49fc0424c0 con 0x7f4a080f54b0 2023-12-26T21:58:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.520+0000 7f4a06ffd700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49f0031090 0x7f49f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:42.522 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.520+0000 7f4a06ffd700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49f0031090 0x7f49f0033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f49f80096a0 tx=0x7f49f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:42.525 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.525+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f49fc01d980 con 0x7f4a080f54b0 2023-12-26T21:58:42.732 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.731+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a08107030 con 0x7f4a080f54b0 2023-12-26T21:58:42.732 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.732+0000 7f49effff700 1 -- 172.21.15.76:0/427710004 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49fc02c090 con 0x7f4a080f54b0 2023-12-26T21:58:42.732 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:42.732 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49f0031090 msgr2=0x7f49f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49f0031090 0x7f49f0033550 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f49f80096a0 tx=0x7f49f8006b40).stop 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 msgr2=0x7f4a081056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a081056a0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f49fc0089f0 tx=0x7f49fc00b040).stop 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 shutdown_connections 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49f0031090 0x7f49f0033550 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 --2- 172.21.15.76:0/427710004 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a080f54b0 0x7f4a081056a0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.733+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 >> 172.21.15.76:0/427710004 conn(0x7f4a080f0c10 msgr2=0x7f4a080f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:42.734 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.734+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 shutdown_connections 2023-12-26T21:58:42.735 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:42.734+0000 7f4a0ddeb700 1 -- 172.21.15.76:0/427710004 wait complete. 2023-12-26T21:58:42.735 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:41.340600+0000 mgr.smithi059.gwkiyc (mgr.14156) 60 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:42.733487+0000 mon.smithi059 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.76:0/427710004' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:44.110 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:44.110 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:45.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:43.340991+0000 mgr.smithi059.gwkiyc (mgr.14156) 61 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:45.352 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 -- 172.21.15.76:0/1263070175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840f5480 msgr2=0x7f67840f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:45.352 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 --2- 172.21.15.76:0/1263070175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840f5480 0x7f67840f58a0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f676c009fb0 tx=0x7f676c009b30).stop 2023-12-26T21:58:45.352 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 -- 172.21.15.76:0/1263070175 shutdown_connections 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 --2- 172.21.15.76:0/1263070175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840f5480 0x7f67840f58a0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 -- 172.21.15.76:0/1263070175 >> 172.21.15.76:0/1263070175 conn(0x7f67840f0be0 msgr2=0x7f67840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 -- 172.21.15.76:0/1263070175 shutdown_connections 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.351+0000 7f6789250700 1 -- 172.21.15.76:0/1263070175 wait complete. 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6789250700 1 Processor -- start 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6789250700 1 -- start start 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6789250700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 0x7f6784109640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:45.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6789250700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6784109b30 con 0x7f67840602c0 2023-12-26T21:58:45.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6782d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 0x7f6784109640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6782d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 0x7f6784109640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52262/0 (socket says 172.21.15.76:52262) 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6782d9d700 1 -- 172.21.15.76:0/4203350266 learned_addr learned my addr 172.21.15.76:0/4203350266 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.352+0000 7f6782d9d700 1 -- 172.21.15.76:0/4203350266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f676c005040 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.353+0000 7f6782d9d700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 0x7f6784109640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f676c0134d0 tx=0x7f676c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.353+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f676c013ee0 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.353+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f676c013680 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.353+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f676c01ee90 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.353+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6784109c70 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.353+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f678410a0a0 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.354+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f676c025060 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.354+0000 7f677b7fe700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6770031040 0x7f6770033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.354+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6784045bb0 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.354+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f676c042370 con 0x7f67840602c0 2023-12-26T21:58:45.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.354+0000 7f678259c700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6770031040 0x7f6770033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:45.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.354+0000 7f678259c700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6770031040 0x7f6770033500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f67740096a0 tx=0x7f6774006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:45.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.359+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f676c01c520 con 0x7f67840602c0 2023-12-26T21:58:45.566 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.566+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67840faf70 con 0x7f67840602c0 2023-12-26T21:58:45.567 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.566+0000 7f677b7fe700 1 -- 172.21.15.76:0/4203350266 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f676c01c340 con 0x7f67840602c0 2023-12-26T21:58:45.567 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:45.567 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:45.568 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6770031040 msgr2=0x7f6770033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6770031040 0x7f6770033500 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f67740096a0 tx=0x7f6774006b40).stop 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 msgr2=0x7f6784109640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 0x7f6784109640 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f676c0134d0 tx=0x7f676c009b30).stop 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 shutdown_connections 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6770031040 0x7f6770033500 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 --2- 172.21.15.76:0/4203350266 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67840602c0 0x7f6784109640 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 >> 172.21.15.76:0/4203350266 conn(0x7f67840f0be0 msgr2=0x7f67840f1770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 shutdown_connections 2023-12-26T21:58:45.569 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:45.568+0000 7f6789250700 1 -- 172.21.15.76:0/4203350266 wait complete. 2023-12-26T21:58:45.570 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:45.466171+0000 mon.smithi059 (mon.0) 143 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T21:58:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:45.466900+0000 mon.smithi059 (mon.0) 144 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T21:58:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:45.568010+0000 mon.smithi059 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.76:0/4203350266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:46.952 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:46.952 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:47.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:45.341290+0000 mgr.smithi059.gwkiyc (mgr.14156) 62 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:48.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- 172.21.15.76:0/3284323039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 msgr2=0x7fcb200f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 --2- 172.21.15.76:0/3284323039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb200f91f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fcb10009fb0 tx=0x7fcb10009b30).stop 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- 172.21.15.76:0/3284323039 shutdown_connections 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 --2- 172.21.15.76:0/3284323039 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb200f91f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- 172.21.15.76:0/3284323039 >> 172.21.15.76:0/3284323039 conn(0x7fcb200f0c10 msgr2=0x7fcb200f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- 172.21.15.76:0/3284323039 shutdown_connections 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- 172.21.15.76:0/3284323039 wait complete. 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 Processor -- start 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- start start 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb20106790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.214+0000 7fcb27b23700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb20106c80 con 0x7fcb20060300 2023-12-26T21:58:48.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb258bf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb20106790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb258bf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb20106790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57472/0 (socket says 172.21.15.76:57472) 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb258bf700 1 -- 172.21.15.76:0/3360715353 learned_addr learned my addr 172.21.15.76:0/3360715353 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb258bf700 1 -- 172.21.15.76:0/3360715353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb10005040 con 0x7fcb20060300 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb258bf700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb20106790 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcb10009f80 tx=0x7fcb1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb10009320 con 0x7fcb20060300 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcb10013680 con 0x7fcb20060300 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb20106e80 con 0x7fcb20060300 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb1001eda0 con 0x7fcb20060300 2023-12-26T21:58:48.217 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.215+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb20107310 con 0x7fcb20060300 2023-12-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.216+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcb1001e420 con 0x7fcb20060300 2023-12-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.216+0000 7fcb167fc700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcb0c031090 0x7fcb0c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.216+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcb10042350 con 0x7fcb20060300 2023-12-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.216+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb20045bb0 con 0x7fcb20060300 2023-12-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.217+0000 7fcb250be700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcb0c031090 0x7fcb0c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:48.218 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.217+0000 7fcb250be700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcb0c031090 0x7fcb0c033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fcb1c0096a0 tx=0x7fcb1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:48.221 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.221+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcb1001e700 con 0x7fcb20060300 2023-12-26T21:58:48.426 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.426+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb20108060 con 0x7fcb20060300 2023-12-26T21:58:48.427 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.426+0000 7fcb167fc700 1 -- 172.21.15.76:0/3360715353 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcb1001cb10 con 0x7fcb20060300 2023-12-26T21:58:48.427 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:48.427 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:48.428 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcb0c031090 msgr2=0x7fcb0c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:48.428 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcb0c031090 0x7fcb0c033550 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fcb1c0096a0 tx=0x7fcb1c006b40).stop 2023-12-26T21:58:48.428 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 msgr2=0x7fcb20106790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb20106790 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fcb10009f80 tx=0x7fcb1000b040).stop 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 shutdown_connections 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcb0c031090 0x7fcb0c033550 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 --2- 172.21.15.76:0/3360715353 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcb20060300 0x7fcb20106790 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 >> 172.21.15.76:0/3360715353 conn(0x7fcb200f0c10 msgr2=0x7fcb200f16e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 shutdown_connections 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:48.428+0000 7fcb27b23700 1 -- 172.21.15.76:0/3360715353 wait complete. 2023-12-26T21:58:48.429 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:47.341572+0000 mgr.smithi059.gwkiyc (mgr.14156) 63 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:48.427932+0000 mon.smithi059 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.76:0/3360715353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:49.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:49.831 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:51.032 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 -- 172.21.15.76:0/1792796472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 msgr2=0x7f45400f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:51.032 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 --2- 172.21.15.76:0/1792796472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400f58d0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f4534009fb0 tx=0x7f4534009b30).stop 2023-12-26T21:58:51.032 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 -- 172.21.15.76:0/1792796472 shutdown_connections 2023-12-26T21:58:51.033 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 --2- 172.21.15.76:0/1792796472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400f58d0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:51.033 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 -- 172.21.15.76:0/1792796472 >> 172.21.15.76:0/1792796472 conn(0x7f45400f0c10 msgr2=0x7f45400f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:51.033 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 -- 172.21.15.76:0/1792796472 shutdown_connections 2023-12-26T21:58:51.033 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 -- 172.21.15.76:0/1792796472 wait complete. 2023-12-26T21:58:51.033 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 Processor -- start 2023-12-26T21:58:51.033 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 -- start start 2023-12-26T21:58:51.034 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.028+0000 7f4544877700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400fd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:51.034 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f4544877700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45400fb910 con 0x7f45400f54b0 2023-12-26T21:58:51.034 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f453e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:51.034 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f453e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400fd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57492/0 (socket says 172.21.15.76:57492) 2023-12-26T21:58:51.034 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f453e59c700 1 -- 172.21.15.76:0/3315758393 learned_addr learned my addr 172.21.15.76:0/3315758393 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:51.034 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f453e59c700 1 -- 172.21.15.76:0/3315758393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4534005040 con 0x7f45400f54b0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f453e59c700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400fd490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4534004db0 tx=0x7f453400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.029+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4534013380 con 0x7f45400f54b0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.030+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45400fbab0 con 0x7f45400f54b0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.030+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4534007e50 con 0x7f45400f54b0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.030+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4534014c30 con 0x7f45400f54b0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.030+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45400fbf40 con 0x7f45400f54b0 2023-12-26T21:58:51.035 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.030+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4534025070 con 0x7f45400f54b0 2023-12-26T21:58:51.036 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.030+0000 7f452effd700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4528031090 0x7f4528033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:51.036 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.031+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4534041da0 con 0x7f45400f54b0 2023-12-26T21:58:51.036 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.031+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45400591e0 con 0x7f45400f54b0 2023-12-26T21:58:51.036 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.031+0000 7f453dd9b700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4528031090 0x7f4528033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:51.036 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.031+0000 7f453dd9b700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4528031090 0x7f4528033550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f45300096a0 tx=0x7f4530006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:51.037 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.035+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f453400f300 con 0x7f45400f54b0 2023-12-26T21:58:51.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.243+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4540045bb0 con 0x7f45400f54b0 2023-12-26T21:58:51.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.244+0000 7f452effd700 1 -- 172.21.15.76:0/3315758393 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f453402d090 con 0x7f45400f54b0 2023-12-26T21:58:51.245 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:51.245 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4528031090 msgr2=0x7f4528033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4528031090 0x7f4528033550 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f45300096a0 tx=0x7f4530006b40).stop 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 msgr2=0x7f45400fd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400fd490 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4534004db0 tx=0x7f453400b040).stop 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 shutdown_connections 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4528031090 0x7f4528033550 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:51.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 --2- 172.21.15.76:0/3315758393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f45400f54b0 0x7f45400fd490 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:51.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 >> 172.21.15.76:0/3315758393 conn(0x7f45400f0c10 msgr2=0x7f454005ff20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:51.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 shutdown_connections 2023-12-26T21:58:51.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:51.245+0000 7f4544877700 1 -- 172.21.15.76:0/3315758393 wait complete. 2023-12-26T21:58:51.247 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:49.341963+0000 mgr.smithi059.gwkiyc (mgr.14156) 64 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:52.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:51 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:51.245411+0000 mon.smithi059 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.76:0/3315758393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:52.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:52.649 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:51.342342+0000 mgr.smithi059.gwkiyc (mgr.14156) 65 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:53.784 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.782+0000 7efc35eb1700 1 -- 172.21.15.76:0/2353020427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 msgr2=0x7efc300f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.782+0000 7efc35eb1700 1 --2- 172.21.15.76:0/2353020427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc300f56f0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7efc24009fb0 tx=0x7efc24009b30).stop 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.782+0000 7efc35eb1700 1 -- 172.21.15.76:0/2353020427 shutdown_connections 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.782+0000 7efc35eb1700 1 --2- 172.21.15.76:0/2353020427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc300f56f0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.782+0000 7efc35eb1700 1 -- 172.21.15.76:0/2353020427 >> 172.21.15.76:0/2353020427 conn(0x7efc300f0a30 msgr2=0x7efc300f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.783+0000 7efc35eb1700 1 -- 172.21.15.76:0/2353020427 shutdown_connections 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.783+0000 7efc35eb1700 1 -- 172.21.15.76:0/2353020427 wait complete. 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.783+0000 7efc35eb1700 1 Processor -- start 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.783+0000 7efc35eb1700 1 -- start start 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.783+0000 7efc35eb1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc30105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.783+0000 7efc35eb1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc30105a00 con 0x7efc300f52d0 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc34eaf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc30105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc34eaf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc30105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57506/0 (socket says 172.21.15.76:57506) 2023-12-26T21:58:53.785 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc34eaf700 1 -- 172.21.15.76:0/636417850 learned_addr learned my addr 172.21.15.76:0/636417850 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc34eaf700 1 -- 172.21.15.76:0/636417850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc24005040 con 0x7efc300f52d0 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc34eaf700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc30105510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7efc240089f0 tx=0x7efc2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc24009320 con 0x7efc300f52d0 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc30105c00 con 0x7efc300f52d0 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efc24013ec0 con 0x7efc300f52d0 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc24014d30 con 0x7efc300f52d0 2023-12-26T21:58:53.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.784+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc30106090 con 0x7efc300f52d0 2023-12-26T21:58:53.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.785+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efc24014420 con 0x7efc300f52d0 2023-12-26T21:58:53.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.785+0000 7efc2d7fa700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc20031090 0x7efc20033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:53.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.785+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc30045bb0 con 0x7efc300f52d0 2023-12-26T21:58:53.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.785+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efc240424c0 con 0x7efc300f52d0 2023-12-26T21:58:53.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.785+0000 7efc2ffff700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc20031090 0x7efc20033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:53.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.786+0000 7efc2ffff700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc20031090 0x7efc20033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7efc1c0096a0 tx=0x7efc1c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:53.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.790+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efc2401d980 con 0x7efc300f52d0 2023-12-26T21:58:53.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.999+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc30106f10 con 0x7efc300f52d0 2023-12-26T21:58:54.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:53.999+0000 7efc2d7fa700 1 -- 172.21.15.76:0/636417850 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc24008620 con 0x7efc300f52d0 2023-12-26T21:58:54.000 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:54.000 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc20031090 msgr2=0x7efc20033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc20031090 0x7efc20033550 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7efc1c0096a0 tx=0x7efc1c006b40).stop 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 msgr2=0x7efc30105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc30105510 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7efc240089f0 tx=0x7efc2400b040).stop 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 shutdown_connections 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc20031090 0x7efc20033550 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 --2- 172.21.15.76:0/636417850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc300f52d0 0x7efc30105510 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 >> 172.21.15.76:0/636417850 conn(0x7efc300f0a30 msgr2=0x7efc300f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 shutdown_connections 2023-12-26T21:58:54.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:54.001+0000 7efc35eb1700 1 -- 172.21.15.76:0/636417850 wait complete. 2023-12-26T21:58:54.003 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:53.342664+0000 mgr.smithi059.gwkiyc (mgr.14156) 66 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:54.001096+0000 mon.smithi059 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.76:0/636417850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:55.391 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:55.391 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:56.609 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 -- 172.21.15.76:0/560642566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 msgr2=0x7f62e00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:56.609 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 --2- 172.21.15.76:0/560642566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e00f69b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f62dc004760 tx=0x7f62dc009ad0).stop 2023-12-26T21:58:56.609 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 -- 172.21.15.76:0/560642566 shutdown_connections 2023-12-26T21:58:56.610 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 --2- 172.21.15.76:0/560642566 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e00f69b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:56.610 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 -- 172.21.15.76:0/560642566 >> 172.21.15.76:0/560642566 conn(0x7f62e00f1cd0 msgr2=0x7f62e00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:56.610 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 -- 172.21.15.76:0/560642566 shutdown_connections 2023-12-26T21:58:56.610 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 -- 172.21.15.76:0/560642566 wait complete. 2023-12-26T21:58:56.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.609+0000 7f62e7d34700 1 Processor -- start 2023-12-26T21:58:56.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e7d34700 1 -- start start 2023-12-26T21:58:56.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e7d34700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e0106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:56.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e7d34700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62e0106cf0 con 0x7f62e00f6590 2023-12-26T21:58:56.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e5ad0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e0106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:56.611 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e5ad0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e0106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46312/0 (socket says 172.21.15.76:46312) 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e5ad0700 1 -- 172.21.15.76:0/2593021098 learned_addr learned my addr 172.21.15.76:0/2593021098 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.610+0000 7f62e5ad0700 1 -- 172.21.15.76:0/2593021098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62dc009d70 con 0x7f62e00f6590 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.611+0000 7f62e5ad0700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e0106800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f62dc004170 tx=0x7f62dc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.611+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62dc00e040 con 0x7f62e00f6590 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.611+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f62dc008ef0 con 0x7f62e00f6590 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.611+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62dc014c90 con 0x7f62e00f6590 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.611+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62e0106ef0 con 0x7f62e00f6590 2023-12-26T21:58:56.612 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.611+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62e0107380 con 0x7f62e00f6590 2023-12-26T21:58:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.612+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f62dc014420 con 0x7f62e00f6590 2023-12-26T21:58:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.612+0000 7f62d67fc700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f62cc0310e0 0x7f62cc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.612+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f62dc0421f0 con 0x7f62e00f6590 2023-12-26T21:58:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.612+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62e0045bb0 con 0x7f62e00f6590 2023-12-26T21:58:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.612+0000 7f62e52cf700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f62cc0310e0 0x7f62cc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:56.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.613+0000 7f62e52cf700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f62cc0310e0 0x7f62cc0335a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f62d00096a0 tx=0x7f62d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:56.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.617+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f62dc026300 con 0x7f62e00f6590 2023-12-26T21:58:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.823+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62e0108180 con 0x7f62e00f6590 2023-12-26T21:58:56.823 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.823+0000 7f62d67fc700 1 -- 172.21.15.76:0/2593021098 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f62dc012b20 con 0x7f62e00f6590 2023-12-26T21:58:56.824 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:56.824 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:56.825 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f62cc0310e0 msgr2=0x7f62cc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f62cc0310e0 0x7f62cc0335a0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f62d00096a0 tx=0x7f62d0006b40).stop 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 msgr2=0x7f62e0106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e0106800 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f62dc004170 tx=0x7f62dc009ad0).stop 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 shutdown_connections 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f62cc0310e0 0x7f62cc0335a0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 --2- 172.21.15.76:0/2593021098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f62e00f6590 0x7f62e0106800 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 >> 172.21.15.76:0/2593021098 conn(0x7f62e00f1cd0 msgr2=0x7f62e00f2810 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 shutdown_connections 2023-12-26T21:58:56.826 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:56.825+0000 7f62e7d34700 1 -- 172.21.15.76:0/2593021098 wait complete. 2023-12-26T21:58:56.827 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:58:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:55.342945+0000 mgr.smithi059.gwkiyc (mgr.14156) 67 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:56.824853+0000 mon.smithi059 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.76:0/2593021098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:58:58.342 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:58:58.342 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:58:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:57.343305+0000 mgr.smithi059.gwkiyc (mgr.14156) 68 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 -- 172.21.15.76:0/2945994740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 msgr2=0x7fbac80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 --2- 172.21.15.76:0/2945994740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac80f58a0 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbab8009fb0 tx=0x7fbab8009b30).stop 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 -- 172.21.15.76:0/2945994740 shutdown_connections 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 --2- 172.21.15.76:0/2945994740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac80f58a0 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 -- 172.21.15.76:0/2945994740 >> 172.21.15.76:0/2945994740 conn(0x7fbac80f0be0 msgr2=0x7fbac80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 -- 172.21.15.76:0/2945994740 shutdown_connections 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.587+0000 7fbacfe98700 1 -- 172.21.15.76:0/2945994740 wait complete. 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacfe98700 1 Processor -- start 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacfe98700 1 -- start start 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacfe98700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac8109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacfe98700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac8060300 con 0x7fbac80f5480 2023-12-26T21:58:59.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacdc34700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac8109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:59.589 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacdc34700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac8109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46322/0 (socket says 172.21.15.76:46322) 2023-12-26T21:58:59.589 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacdc34700 1 -- 172.21.15.76:0/314649133 learned_addr learned my addr 172.21.15.76:0/314649133 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:58:59.589 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.588+0000 7fbacdc34700 1 -- 172.21.15.76:0/314649133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab8005040 con 0x7fbac80f5480 2023-12-26T21:58:59.589 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.589+0000 7fbacdc34700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac8109510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbab80089f0 tx=0x7fbab800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.589+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab8009320 con 0x7fbac80f5480 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.589+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac8060500 con 0x7fbac80f5480 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.589+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbab8013ec0 con 0x7fbac80f5480 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.589+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbab8014d30 con 0x7fbac80f5480 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.589+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac8109bc0 con 0x7fbac80f5480 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.590+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbab80263a0 con 0x7fbac80f5480 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.590+0000 7fbabe7fc700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbab4031090 0x7fbab4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:58:59.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.590+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac80591e0 con 0x7fbac80f5480 2023-12-26T21:58:59.591 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.590+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbab8042450 con 0x7fbac80f5480 2023-12-26T21:58:59.591 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.590+0000 7fbacd433700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbab4031090 0x7fbab4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:58:59.591 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.591+0000 7fbacd433700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbab4031090 0x7fbab4033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbac40096a0 tx=0x7fbac4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:58:59.594 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.595+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbab801cb10 con 0x7fbac80f5480 2023-12-26T21:58:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.801+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbac80faf70 con 0x7fbac80f5480 2023-12-26T21:58:59.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.802+0000 7fbabe7fc700 1 -- 172.21.15.76:0/314649133 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbab801cb10 con 0x7fbac80f5480 2023-12-26T21:58:59.802 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:58:59.802 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:58:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbab4031090 msgr2=0x7fbab4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbab4031090 0x7fbab4033550 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fbac40096a0 tx=0x7fbac4006b40).stop 2023-12-26T21:58:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 msgr2=0x7fbac8109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:58:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac8109510 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7fbab80089f0 tx=0x7fbab800b040).stop 2023-12-26T21:58:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 shutdown_connections 2023-12-26T21:58:59.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbab4031090 0x7fbab4033550 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 --2- 172.21.15.76:0/314649133 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbac80f5480 0x7fbac8109510 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:58:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 >> 172.21.15.76:0/314649133 conn(0x7fbac80f0be0 msgr2=0x7fbac80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:58:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 shutdown_connections 2023-12-26T21:58:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:58:59.804+0000 7fbacfe98700 1 -- 172.21.15.76:0/314649133 wait complete. 2023-12-26T21:58:59.804 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:00.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:58:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:58:59.802800+0000 mon.smithi059 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.76:0/314649133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:01.189 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:01.189 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:58:59.343554+0000 mgr.smithi059.gwkiyc (mgr.14156) 69 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- 172.21.15.76:0/3476913884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 msgr2=0x7fbd9c0f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 --2- 172.21.15.76:0/3476913884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f8fb0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbd98009fb0 tx=0x7fbd98009b30).stop 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- 172.21.15.76:0/3476913884 shutdown_connections 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 --2- 172.21.15.76:0/3476913884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f8fb0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- 172.21.15.76:0/3476913884 >> 172.21.15.76:0/3476913884 conn(0x7fbd9c0f0a30 msgr2=0x7fbd9c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- 172.21.15.76:0/3476913884 shutdown_connections 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- 172.21.15.76:0/3476913884 wait complete. 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 Processor -- start 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- start start 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f6a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:02.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.413+0000 7fbda24d1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbd9c0f4eb0 con 0x7fbd9c060360 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbda14cf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f6a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbda14cf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f6a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46354/0 (socket says 172.21.15.76:46354) 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbda14cf700 1 -- 172.21.15.76:0/1019022293 learned_addr learned my addr 172.21.15.76:0/1019022293 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbda14cf700 1 -- 172.21.15.76:0/1019022293 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbd98005040 con 0x7fbd9c060360 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbda14cf700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f6a30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbd98009f80 tx=0x7fbd9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd98009320 con 0x7fbd9c060360 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbd9c0f5050 con 0x7fbd9c060360 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbd98013680 con 0x7fbd9c060360 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.414+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbd9801eda0 con 0x7fbd9c060360 2023-12-26T21:59:02.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.415+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbd9c0f54e0 con 0x7fbd9c060360 2023-12-26T21:59:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.415+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbd9801e420 con 0x7fbd9c060360 2023-12-26T21:59:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.415+0000 7fbd91ffb700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbd88031090 0x7fbd88033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.415+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbd9c0591e0 con 0x7fbd9c060360 2023-12-26T21:59:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.416+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbd98042360 con 0x7fbd9c060360 2023-12-26T21:59:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.416+0000 7fbda0cce700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbd88031090 0x7fbd88033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:02.416 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.416+0000 7fbda0cce700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbd88031090 0x7fbd88033550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbd8c0096a0 tx=0x7fbd8c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:02.420 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.421+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbd9801d380 con 0x7fbd9c060360 2023-12-26T21:59:02.625 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.625+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbd9c0fbeb0 con 0x7fbd9c060360 2023-12-26T21:59:02.625 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.626+0000 7fbd91ffb700 1 -- 172.21.15.76:0/1019022293 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbd98004490 con 0x7fbd9c060360 2023-12-26T21:59:02.625 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:02.628 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.627+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbd88031090 msgr2=0x7fbd88033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.627+0000 7fbda24d1700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbd88031090 0x7fbd88033550 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fbd8c0096a0 tx=0x7fbd8c006b40).stop 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.627+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 msgr2=0x7fbd9c0f6a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.627+0000 7fbda24d1700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f6a30 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fbd98009f80 tx=0x7fbd9800b040).stop 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.628+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 shutdown_connections 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.628+0000 7fbda24d1700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbd88031090 0x7fbd88033550 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:02.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.628+0000 7fbda24d1700 1 --2- 172.21.15.76:0/1019022293 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbd9c060360 0x7fbd9c0f6a30 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:02.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.628+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 >> 172.21.15.76:0/1019022293 conn(0x7fbd9c0f0a30 msgr2=0x7fbd9c0f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:02.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.628+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 shutdown_connections 2023-12-26T21:59:02.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:02.628+0000 7fbda24d1700 1 -- 172.21.15.76:0/1019022293 wait complete. 2023-12-26T21:59:02.630 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:01.343890+0000 mgr.smithi059.gwkiyc (mgr.14156) 70 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:02.626580+0000 mon.smithi059 (mon.0) 151 : audit [DBG] from='client.? 172.21.15.76:0/1019022293' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:04.027 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:04.027 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:03.344266+0000 mgr.smithi059.gwkiyc (mgr.14156) 71 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- 172.21.15.76:0/4115214368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 msgr2=0x7efc640f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 --2- 172.21.15.76:0/4115214368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc640f58a0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7efc54009fb0 tx=0x7efc54009b30).stop 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- 172.21.15.76:0/4115214368 shutdown_connections 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 --2- 172.21.15.76:0/4115214368 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc640f58a0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- 172.21.15.76:0/4115214368 >> 172.21.15.76:0/4115214368 conn(0x7efc640f0be0 msgr2=0x7efc640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- 172.21.15.76:0/4115214368 shutdown_connections 2023-12-26T21:59:05.296 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- 172.21.15.76:0/4115214368 wait complete. 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 Processor -- start 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- start start 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc641056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.297+0000 7efc68eb1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc64105be0 con 0x7efc640f5480 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc6259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc641056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc6259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc641056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46372/0 (socket says 172.21.15.76:46372) 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc6259c700 1 -- 172.21.15.76:0/2362640481 learned_addr learned my addr 172.21.15.76:0/2362640481 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc6259c700 1 -- 172.21.15.76:0/2362640481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54005040 con 0x7efc640f5480 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc6259c700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc641056f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7efc540089f0 tx=0x7efc5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc54009320 con 0x7efc640f5480 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc64105de0 con 0x7efc640f5480 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efc54013ec0 con 0x7efc640f5480 2023-12-26T21:59:05.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc54014d30 con 0x7efc640f5480 2023-12-26T21:59:05.299 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.298+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc64106270 con 0x7efc640f5480 2023-12-26T21:59:05.299 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.299+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efc540263a0 con 0x7efc640f5480 2023-12-26T21:59:05.299 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.299+0000 7efc52ffd700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc4c031090 0x7efc4c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:05.299 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.299+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc64045bb0 con 0x7efc640f5480 2023-12-26T21:59:05.300 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.299+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efc54042440 con 0x7efc640f5480 2023-12-26T21:59:05.300 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.299+0000 7efc61d9b700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc4c031090 0x7efc4c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:05.300 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.300+0000 7efc61d9b700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc4c031090 0x7efc4c033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efc580096a0 tx=0x7efc58006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:05.303 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.304+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efc5401cb10 con 0x7efc640f5480 2023-12-26T21:59:05.509 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.510+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc64107070 con 0x7efc640f5480 2023-12-26T21:59:05.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.511+0000 7efc52ffd700 1 -- 172.21.15.76:0/2362640481 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc5403b030 con 0x7efc640f5480 2023-12-26T21:59:05.510 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:05.510 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc4c031090 msgr2=0x7efc4c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc4c031090 0x7efc4c033550 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7efc580096a0 tx=0x7efc58006b40).stop 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 msgr2=0x7efc641056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc641056f0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7efc540089f0 tx=0x7efc5400b040).stop 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 shutdown_connections 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc4c031090 0x7efc4c033550 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 --2- 172.21.15.76:0/2362640481 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc640f5480 0x7efc641056f0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 >> 172.21.15.76:0/2362640481 conn(0x7efc640f0be0 msgr2=0x7efc640f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 shutdown_connections 2023-12-26T21:59:05.512 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:05.512+0000 7efc68eb1700 1 -- 172.21.15.76:0/2362640481 wait complete. 2023-12-26T21:59:05.513 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:06.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:05 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:05.511318+0000 mon.smithi059 (mon.0) 152 : audit [DBG] from='client.? 172.21.15.76:0/2362640481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:06.905 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:06.906 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:05.344546+0000 mgr.smithi059.gwkiyc (mgr.14156) 72 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:08.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/3254142101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 msgr2=0x7f9fac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:08.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 --2- 172.21.15.76:0/3254142101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac0f69b0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f9fa4009a00 tx=0x7f9fa4004cb0).stop 2023-12-26T21:59:08.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/3254142101 shutdown_connections 2023-12-26T21:59:08.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 --2- 172.21.15.76:0/3254142101 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac0f69b0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/3254142101 >> 172.21.15.76:0/3254142101 conn(0x7f9fac0f1cd0 msgr2=0x7f9fac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/3254142101 shutdown_connections 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.144+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/3254142101 wait complete. 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9fb0ef7700 1 Processor -- start 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9fb0ef7700 1 -- start start 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9fb0ef7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac106770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:08.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9fb0ef7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fac106c60 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9faa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac106770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9faa59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac106770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:42664/0 (socket says 172.21.15.76:42664) 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.145+0000 7f9faa59c700 1 -- 172.21.15.76:0/297370952 learned_addr learned my addr 172.21.15.76:0/297370952 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9faa59c700 1 -- 172.21.15.76:0/297370952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa4009640 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9faa59c700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac106770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f9fa4009610 tx=0x7f9fa4004cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa40092e0 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9fa4007d50 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9fa4003de0 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fac106e60 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.146+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fac1072f0 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.147+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9fa4003650 con 0x7f9fac0f6590 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.147+0000 7f9f9affd700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9f94031090 0x7f9f94033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:08.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.147+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fac045bb0 con 0x7f9fac0f6590 2023-12-26T21:59:08.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.147+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9fa4042d60 con 0x7f9fac0f6590 2023-12-26T21:59:08.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.147+0000 7f9fa9d9b700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9f94031090 0x7f9f94033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:08.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.147+0000 7f9fa9d9b700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9f94031090 0x7f9f94033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9fa00096a0 tx=0x7f9fa0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:08.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.152+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9fa4003450 con 0x7f9fac0f6590 2023-12-26T21:59:08.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.357+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9fac108170 con 0x7f9fac0f6590 2023-12-26T21:59:08.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.358+0000 7f9f9affd700 1 -- 172.21.15.76:0/297370952 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9fa4040090 con 0x7f9fac0f6590 2023-12-26T21:59:08.358 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:08.358 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:08.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9f94031090 msgr2=0x7f9f94033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:08.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9f94031090 0x7f9f94033550 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f9fa00096a0 tx=0x7f9fa0006b40).stop 2023-12-26T21:59:08.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 msgr2=0x7f9fac106770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:08.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac106770 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f9fa4009610 tx=0x7f9fa4004cb0).stop 2023-12-26T21:59:08.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 shutdown_connections 2023-12-26T21:59:08.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9f94031090 0x7f9f94033550 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:08.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 --2- 172.21.15.76:0/297370952 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9fac0f6590 0x7f9fac106770 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:08.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 >> 172.21.15.76:0/297370952 conn(0x7f9fac0f1cd0 msgr2=0x7f9fac0f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:08.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 shutdown_connections 2023-12-26T21:59:08.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:08.360+0000 7f9fb0ef7700 1 -- 172.21.15.76:0/297370952 wait complete. 2023-12-26T21:59:08.360 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:07.344899+0000 mgr.smithi059.gwkiyc (mgr.14156) 73 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:08.358371+0000 mon.smithi059 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.76:0/297370952' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:09.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:09.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 -- 172.21.15.76:0/2582208472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 msgr2=0x7f949c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 --2- 172.21.15.76:0/2582208472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0f58a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9490009fb0 tx=0x7f9490009b30).stop 2023-12-26T21:59:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 -- 172.21.15.76:0/2582208472 shutdown_connections 2023-12-26T21:59:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 --2- 172.21.15.76:0/2582208472 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0f58a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 -- 172.21.15.76:0/2582208472 >> 172.21.15.76:0/2582208472 conn(0x7f949c0f0be0 msgr2=0x7f949c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:10.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 -- 172.21.15.76:0/2582208472 shutdown_connections 2023-12-26T21:59:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.951+0000 7f94a30f5700 1 -- 172.21.15.76:0/2582208472 wait complete. 2023-12-26T21:59:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a30f5700 1 Processor -- start 2023-12-26T21:59:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a30f5700 1 -- start start 2023-12-26T21:59:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a30f5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a30f5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f949c0fb8e0 con 0x7f949c0f5480 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a0e91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a0e91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:42678/0 (socket says 172.21.15.76:42678) 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.952+0000 7f94a0e91700 1 -- 172.21.15.76:0/4041198993 learned_addr learned my addr 172.21.15.76:0/4041198993 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94a0e91700 1 -- 172.21.15.76:0/4041198993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9490005040 con 0x7f949c0f5480 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94a0e91700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0fd460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f94900089f0 tx=0x7f949000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9490009320 con 0x7f949c0f5480 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f949c0fba80 con 0x7f949c0f5480 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9490013ec0 con 0x7f949c0f5480 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9490014d30 con 0x7f949c0f5480 2023-12-26T21:59:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.953+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f949c0fbf10 con 0x7f949c0f5480 2023-12-26T21:59:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.954+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f94900263a0 con 0x7f949c0f5480 2023-12-26T21:59:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.954+0000 7f94997fa700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f948c031090 0x7f948c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:10.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.954+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f949c0591e0 con 0x7f949c0f5480 2023-12-26T21:59:10.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.954+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9490042380 con 0x7f949c0f5480 2023-12-26T21:59:10.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.954+0000 7f949bfff700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f948c031090 0x7f948c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:10.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.954+0000 7f949bfff700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f948c031090 0x7f948c033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f94880096a0 tx=0x7f9488006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:10.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:10.959+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9490012900 con 0x7f949c0f5480 2023-12-26T21:59:11.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.170+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f949c0faf70 con 0x7f949c0f5480 2023-12-26T21:59:11.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.171+0000 7f94997fa700 1 -- 172.21.15.76:0/4041198993 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9490012330 con 0x7f949c0f5480 2023-12-26T21:59:11.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:11.171 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.172+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f948c031090 msgr2=0x7f948c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.172+0000 7f94a30f5700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f948c031090 0x7f948c033550 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f94880096a0 tx=0x7f9488006b40).stop 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.172+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 msgr2=0x7f949c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.172+0000 7f94a30f5700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0fd460 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f94900089f0 tx=0x7f949000b040).stop 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.173+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 shutdown_connections 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.173+0000 7f94a30f5700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f948c031090 0x7f948c033550 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.173+0000 7f94a30f5700 1 --2- 172.21.15.76:0/4041198993 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f949c0f5480 0x7f949c0fd460 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:11.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.173+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 >> 172.21.15.76:0/4041198993 conn(0x7f949c0f0be0 msgr2=0x7f949c05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:11.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.173+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 shutdown_connections 2023-12-26T21:59:11.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:11.173+0000 7f94a30f5700 1 -- 172.21.15.76:0/4041198993 wait complete. 2023-12-26T21:59:11.173 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:09.345267+0000 mgr.smithi059.gwkiyc (mgr.14156) 74 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:12.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:11 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:11.171635+0000 mon.smithi059 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.76:0/4041198993' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:12.570 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:12.571 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:13.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:11.345577+0000 mgr.smithi059.gwkiyc (mgr.14156) 75 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.792+0000 7f2839e27700 1 -- 172.21.15.76:0/3399036369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 msgr2=0x7f28340f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.792+0000 7f2839e27700 1 --2- 172.21.15.76:0/3399036369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28340f58a0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f2824009fb0 tx=0x7f2824009b30).stop 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.792+0000 7f2839e27700 1 -- 172.21.15.76:0/3399036369 shutdown_connections 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.792+0000 7f2839e27700 1 --2- 172.21.15.76:0/3399036369 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28340f58a0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.792+0000 7f2839e27700 1 -- 172.21.15.76:0/3399036369 >> 172.21.15.76:0/3399036369 conn(0x7f28340f0be0 msgr2=0x7f28340f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.793+0000 7f2839e27700 1 -- 172.21.15.76:0/3399036369 shutdown_connections 2023-12-26T21:59:13.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.793+0000 7f2839e27700 1 -- 172.21.15.76:0/3399036369 wait complete. 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.793+0000 7f2839e27700 1 Processor -- start 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.793+0000 7f2839e27700 1 -- start start 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f2839e27700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28341056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f2839e27700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2834105be0 con 0x7f28340f5480 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f28337fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28341056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f28337fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28341056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:42692/0 (socket says 172.21.15.76:42692) 2023-12-26T21:59:13.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f28337fe700 1 -- 172.21.15.76:0/2565872803 learned_addr learned my addr 172.21.15.76:0/2565872803 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f28337fe700 1 -- 172.21.15.76:0/2565872803 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2824005040 con 0x7f28340f5480 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f28337fe700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28341056f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f28240089f0 tx=0x7f282400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.794+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2824009320 con 0x7f28340f5480 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.795+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2834105de0 con 0x7f28340f5480 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.795+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2824013ec0 con 0x7f28340f5480 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.795+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2824014d30 con 0x7f28340f5480 2023-12-26T21:59:13.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.795+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2834106270 con 0x7f28340f5480 2023-12-26T21:59:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.795+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f28240263a0 con 0x7f28340f5480 2023-12-26T21:59:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.796+0000 7f281bfff700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f281c0310e0 0x7f281c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.796+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2824042490 con 0x7f28340f5480 2023-12-26T21:59:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.796+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2834045bb0 con 0x7f28340f5480 2023-12-26T21:59:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.796+0000 7f2832ffd700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f281c0310e0 0x7f281c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:13.797 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.796+0000 7f2832ffd700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f281c0310e0 0x7f281c0335a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f28280096a0 tx=0x7f2828006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:13.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:13.801+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f28240266d0 con 0x7f28340f5480 2023-12-26T21:59:14.007 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.007+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28340faf70 con 0x7f28340f5480 2023-12-26T21:59:14.007 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.007+0000 7f281bfff700 1 -- 172.21.15.76:0/2565872803 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2824012330 con 0x7f28340f5480 2023-12-26T21:59:14.008 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:14.008 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:14.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f281c0310e0 msgr2=0x7f281c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:14.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f281c0310e0 0x7f281c0335a0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f28280096a0 tx=0x7f2828006b40).stop 2023-12-26T21:59:14.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 msgr2=0x7f28341056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:14.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28341056f0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f28240089f0 tx=0x7f282400b040).stop 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 shutdown_connections 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f281c0310e0 0x7f281c0335a0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 --2- 172.21.15.76:0/2565872803 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28340f5480 0x7f28341056f0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 >> 172.21.15.76:0/2565872803 conn(0x7f28340f0be0 msgr2=0x7f28340f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 shutdown_connections 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:14.009+0000 7f2839e27700 1 -- 172.21.15.76:0/2565872803 wait complete. 2023-12-26T21:59:14.010 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:13.345849+0000 mgr.smithi059.gwkiyc (mgr.14156) 76 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:14.008795+0000 mon.smithi059 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.76:0/2565872803' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:15.422 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:15.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:16.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 -- 172.21.15.76:0/2442832336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 msgr2=0x7f1a840f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 --2- 172.21.15.76:0/2442832336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a840f9160 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f1a6c009fb0 tx=0x7f1a6c009b30).stop 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 -- 172.21.15.76:0/2442832336 shutdown_connections 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 --2- 172.21.15.76:0/2442832336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a840f9160 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 -- 172.21.15.76:0/2442832336 >> 172.21.15.76:0/2442832336 conn(0x7f1a840f0be0 msgr2=0x7f1a840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 -- 172.21.15.76:0/2442832336 shutdown_connections 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.625+0000 7f1a89de3700 1 -- 172.21.15.76:0/2442832336 wait complete. 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a89de3700 1 Processor -- start 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a89de3700 1 -- start start 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a89de3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a84106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a89de3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a84106c40 con 0x7f1a84060360 2023-12-26T21:59:16.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a84106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a837fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a84106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:35446/0 (socket says 172.21.15.76:35446) 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.626+0000 7f1a837fe700 1 -- 172.21.15.76:0/3831462068 learned_addr learned my addr 172.21.15.76:0/3831462068 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a837fe700 1 -- 172.21.15.76:0/3831462068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a6c005040 con 0x7f1a84060360 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a837fe700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a84106750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1a6c009f80 tx=0x7f1a6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a6c009320 con 0x7f1a84060360 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a84106e40 con 0x7f1a84060360 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1a6c013680 con 0x7f1a84060360 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a6c01eda0 con 0x7f1a84060360 2023-12-26T21:59:16.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.627+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a841072d0 con 0x7f1a84060360 2023-12-26T21:59:16.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.628+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1a6c014420 con 0x7f1a84060360 2023-12-26T21:59:16.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.628+0000 7f1a7bfff700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1a70031090 0x7f1a70033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:16.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.628+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1a6c0423a0 con 0x7f1a84060360 2023-12-26T21:59:16.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.628+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a840591e0 con 0x7f1a84060360 2023-12-26T21:59:16.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.628+0000 7f1a82ffd700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1a70031090 0x7f1a70033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:16.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.628+0000 7f1a82ffd700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1a70031090 0x7f1a70033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1a740096a0 tx=0x7f1a74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:16.633 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.633+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1a6c014700 con 0x7f1a84060360 2023-12-26T21:59:16.841 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.841+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a84045bb0 con 0x7f1a84060360 2023-12-26T21:59:16.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.842+0000 7f1a7bfff700 1 -- 172.21.15.76:0/3831462068 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1a6c01c320 con 0x7f1a84060360 2023-12-26T21:59:16.842 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:16.842 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1a70031090 msgr2=0x7f1a70033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1a70031090 0x7f1a70033550 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f1a740096a0 tx=0x7f1a74006b40).stop 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 msgr2=0x7f1a84106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a84106750 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1a6c009f80 tx=0x7f1a6c00b040).stop 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 shutdown_connections 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1a70031090 0x7f1a70033550 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:16.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 --2- 172.21.15.76:0/3831462068 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1a84060360 0x7f1a84106750 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.843+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 >> 172.21.15.76:0/3831462068 conn(0x7f1a840f0be0 msgr2=0x7f1a840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.844+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 shutdown_connections 2023-12-26T21:59:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:16.844+0000 7f1a89de3700 1 -- 172.21.15.76:0/3831462068 wait complete. 2023-12-26T21:59:16.845 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:17.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:15.346228+0000 mgr.smithi059.gwkiyc (mgr.14156) 77 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:17.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:16.843341+0000 mon.smithi059 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.76:0/3831462068' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:18.250 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:18.251 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:19.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:17.346593+0000 mgr.smithi059.gwkiyc (mgr.14156) 78 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 -- 172.21.15.76:0/697101020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 msgr2=0x7f85840f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 --2- 172.21.15.76:0/697101020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85840f4c70 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f8578009fb0 tx=0x7f8578009b30).stop 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 -- 172.21.15.76:0/697101020 shutdown_connections 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 --2- 172.21.15.76:0/697101020 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85840f4c70 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 -- 172.21.15.76:0/697101020 >> 172.21.15.76:0/697101020 conn(0x7f85840f0390 msgr2=0x7f85840f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 -- 172.21.15.76:0/697101020 shutdown_connections 2023-12-26T21:59:19.396 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.395+0000 7f8589a43700 1 -- 172.21.15.76:0/697101020 wait complete. 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.396+0000 7f8589a43700 1 Processor -- start 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.396+0000 7f8589a43700 1 -- start start 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8589a43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85841067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8589a43700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8584106c90 con 0x7f85840f4850 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8582ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85841067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8582ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85841067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:35464/0 (socket says 172.21.15.76:35464) 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8582ffd700 1 -- 172.21.15.76:0/914832115 learned_addr learned my addr 172.21.15.76:0/914832115 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8582ffd700 1 -- 172.21.15.76:0/914832115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8578005040 con 0x7f85840f4850 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.397+0000 7f8582ffd700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85841067a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f85780089f0 tx=0x7f857800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:19.398 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.398+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8578009320 con 0x7f85840f4850 2023-12-26T21:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.398+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8584106e90 con 0x7f85840f4850 2023-12-26T21:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.398+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8578013ec0 con 0x7f85840f4850 2023-12-26T21:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.398+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8578014d30 con 0x7f85840f4850 2023-12-26T21:59:19.399 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.398+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8584107320 con 0x7f85840f4850 2023-12-26T21:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.398+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8578014420 con 0x7f85840f4850 2023-12-26T21:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.399+0000 7f856b7fe700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f856c0310e0 0x7f856c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.399+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8578042470 con 0x7f85840f4850 2023-12-26T21:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.399+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8584045bb0 con 0x7f85840f4850 2023-12-26T21:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.399+0000 7f85827fc700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f856c0310e0 0x7f856c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:19.400 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.399+0000 7f85827fc700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f856c0310e0 0x7f856c0335a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f85740096a0 tx=0x7f8574006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.404+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8578014750 con 0x7f85840f4850 2023-12-26T21:59:19.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.615+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f85840fc080 con 0x7f85840f4850 2023-12-26T21:59:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.617+0000 7f856b7fe700 1 -- 172.21.15.76:0/914832115 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f857801d380 con 0x7f85840f4850 2023-12-26T21:59:19.616 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:19.616 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f856c0310e0 msgr2=0x7f856c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f856c0310e0 0x7f856c0335a0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f85740096a0 tx=0x7f8574006b40).stop 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 msgr2=0x7f85841067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85841067a0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f85780089f0 tx=0x7f857800b040).stop 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 shutdown_connections 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f856c0310e0 0x7f856c0335a0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 --2- 172.21.15.76:0/914832115 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85840f4850 0x7f85841067a0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 >> 172.21.15.76:0/914832115 conn(0x7f85840f0390 msgr2=0x7f85840f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 shutdown_connections 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:19.618+0000 7f8589a43700 1 -- 172.21.15.76:0/914832115 wait complete. 2023-12-26T21:59:19.621 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:20.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:19 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:19.617495+0000 mon.smithi059 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.76:0/914832115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:21.021 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:21.021 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:19.346944+0000 mgr.smithi059.gwkiyc (mgr.14156) 79 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 -- 172.21.15.76:0/3560781176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 msgr2=0x7fccb40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 --2- 172.21.15.76:0/3560781176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb40f58a0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fcca4009fb0 tx=0x7fcca4009b30).stop 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 -- 172.21.15.76:0/3560781176 shutdown_connections 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 --2- 172.21.15.76:0/3560781176 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb40f58a0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 -- 172.21.15.76:0/3560781176 >> 172.21.15.76:0/3560781176 conn(0x7fccb40f0be0 msgr2=0x7fccb40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 -- 172.21.15.76:0/3560781176 shutdown_connections 2023-12-26T21:59:22.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 -- 172.21.15.76:0/3560781176 wait complete. 2023-12-26T21:59:22.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.240+0000 7fccb9f70700 1 Processor -- start 2023-12-26T21:59:22.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb9f70700 1 -- start start 2023-12-26T21:59:22.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb9f70700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb41056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:22.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb9f70700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb4105be0 con 0x7fccb40f5480 2023-12-26T21:59:22.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb41056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:22.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb37fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb41056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:35484/0 (socket says 172.21.15.76:35484) 2023-12-26T21:59:22.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb37fe700 1 -- 172.21.15.76:0/3104897942 learned_addr learned my addr 172.21.15.76:0/3104897942 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:22.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.241+0000 7fccb37fe700 1 -- 172.21.15.76:0/3104897942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca4005040 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.242+0000 7fccb37fe700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb41056f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fcca4000c00 tx=0x7fcca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.242+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca40095f0 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.242+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcca4007e50 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.242+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb4105de0 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.242+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca4014e70 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.242+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb4106270 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.243+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcca400f530 con 0x7fccb40f5480 2023-12-26T21:59:22.243 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.243+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb4045bb0 con 0x7fccb40f5480 2023-12-26T21:59:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.243+0000 7fcc9bfff700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcc9c031090 0x7fcc9c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.243+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcca40439a0 con 0x7fccb40f5480 2023-12-26T21:59:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.243+0000 7fccb2ffd700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcc9c031090 0x7fcc9c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.244+0000 7fccb2ffd700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcc9c031090 0x7fcc9c033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fcca80096a0 tx=0x7fcca8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:22.248 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.248+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcca4030420 con 0x7fccb40f5480 2023-12-26T21:59:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.454+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccb40faf70 con 0x7fccb40f5480 2023-12-26T21:59:22.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.455+0000 7fcc9bfff700 1 -- 172.21.15.76:0/3104897942 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fccb40faf70 con 0x7fccb40f5480 2023-12-26T21:59:22.455 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:22.455 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcc9c031090 msgr2=0x7fcc9c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcc9c031090 0x7fcc9c033550 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fcca80096a0 tx=0x7fcca8006b40).stop 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 msgr2=0x7fccb41056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb41056f0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fcca4000c00 tx=0x7fcca400b040).stop 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 shutdown_connections 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcc9c031090 0x7fcc9c033550 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 --2- 172.21.15.76:0/3104897942 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fccb40f5480 0x7fccb41056f0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 >> 172.21.15.76:0/3104897942 conn(0x7fccb40f0be0 msgr2=0x7fccb40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 shutdown_connections 2023-12-26T21:59:22.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:22.456+0000 7fccb9f70700 1 -- 172.21.15.76:0/3104897942 wait complete. 2023-12-26T21:59:22.458 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:21.347269+0000 mgr.smithi059.gwkiyc (mgr.14156) 80 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:22.456453+0000 mon.smithi059 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.76:0/3104897942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:23.853 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:23.853 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:25.022 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- 172.21.15.76:0/3711874184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 msgr2=0x7faaf80f4d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 --2- 172.21.15.76:0/3711874184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f4d10 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7faaec009fb0 tx=0x7faaec009b30).stop 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- 172.21.15.76:0/3711874184 shutdown_connections 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 --2- 172.21.15.76:0/3711874184 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f4d10 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- 172.21.15.76:0/3711874184 >> 172.21.15.76:0/3711874184 conn(0x7faaf80f0390 msgr2=0x7faaf80f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- 172.21.15.76:0/3711874184 shutdown_connections 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- 172.21.15.76:0/3711874184 wait complete. 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 Processor -- start 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- start start 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f74e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.021+0000 7faafcfcd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faaf80f5960 con 0x7faaf80f48f0 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faaf659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f74e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faaf659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f74e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:35494/0 (socket says 172.21.15.76:35494) 2023-12-26T21:59:25.023 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faaf659c700 1 -- 172.21.15.76:0/1511621201 learned_addr learned my addr 172.21.15.76:0/1511621201 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faaf659c700 1 -- 172.21.15.76:0/1511621201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faaec005040 con 0x7faaf80f48f0 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faaf659c700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f74e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7faaec009f80 tx=0x7faaec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaec009320 con 0x7faaf80f48f0 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faaec013680 con 0x7faaf80f48f0 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faaec01eda0 con 0x7faaf80f48f0 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faaf80f5b00 con 0x7faaf80f48f0 2023-12-26T21:59:25.024 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.022+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faaf80f5f90 con 0x7faaf80f48f0 2023-12-26T21:59:25.025 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.023+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7faaec01e420 con 0x7faaf80f48f0 2023-12-26T21:59:25.025 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.023+0000 7faae6ffd700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faae0031090 0x7faae0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:25.025 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.023+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faaf80591e0 con 0x7faaf80f48f0 2023-12-26T21:59:25.025 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.023+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faaec042330 con 0x7faaf80f48f0 2023-12-26T21:59:25.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.024+0000 7faaf5d9b700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faae0031090 0x7faae0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:25.026 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.024+0000 7faaf5d9b700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faae0031090 0x7faae0033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7faae80096a0 tx=0x7faae8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:25.028 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.028+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faaec01ccf0 con 0x7faaf80f48f0 2023-12-26T21:59:25.238 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.238+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faaf8045bb0 con 0x7faaf80f48f0 2023-12-26T21:59:25.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.239+0000 7faae6ffd700 1 -- 172.21.15.76:0/1511621201 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faaec03e030 con 0x7faaf80f48f0 2023-12-26T21:59:25.239 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:25.240 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faae0031090 msgr2=0x7faae0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faae0031090 0x7faae0033550 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7faae80096a0 tx=0x7faae8006b40).stop 2023-12-26T21:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 msgr2=0x7faaf80f74e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:25.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f74e0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7faaec009f80 tx=0x7faaec00b040).stop 2023-12-26T21:59:25.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 shutdown_connections 2023-12-26T21:59:25.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faae0031090 0x7faae0033550 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:25.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 --2- 172.21.15.76:0/1511621201 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faaf80f48f0 0x7faaf80f74e0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:25.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 >> 172.21.15.76:0/1511621201 conn(0x7faaf80f0390 msgr2=0x7faaf80f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:25.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 shutdown_connections 2023-12-26T21:59:25.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:25.240+0000 7faafcfcd700 1 -- 172.21.15.76:0/1511621201 wait complete. 2023-12-26T21:59:25.243 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:23.347630+0000 mgr.smithi059.gwkiyc (mgr.14156) 81 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:26.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:25.240065+0000 mon.smithi059 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.76:0/1511621201' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:26.635 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:26.635 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:27.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:25.347952+0000 mgr.smithi059.gwkiyc (mgr.14156) 82 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.839+0000 7f3892812700 1 -- 172.21.15.76:0/426754923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 msgr2=0x7f388c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.839+0000 7f3892812700 1 --2- 172.21.15.76:0/426754923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c0f69b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f387c004760 tx=0x7f387c009ad0).stop 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.839+0000 7f3892812700 1 -- 172.21.15.76:0/426754923 shutdown_connections 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.839+0000 7f3892812700 1 --2- 172.21.15.76:0/426754923 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c0f69b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.840+0000 7f3892812700 1 -- 172.21.15.76:0/426754923 >> 172.21.15.76:0/426754923 conn(0x7f388c0f1cd0 msgr2=0x7f388c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.840+0000 7f3892812700 1 -- 172.21.15.76:0/426754923 shutdown_connections 2023-12-26T21:59:27.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.840+0000 7f3892812700 1 -- 172.21.15.76:0/426754923 wait complete. 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.840+0000 7f3892812700 1 Processor -- start 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.840+0000 7f3892812700 1 -- start start 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.841+0000 7f3892812700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.841+0000 7f3892812700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f388c106cb0 con 0x7f388c0f6590 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.841+0000 7f388bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.841+0000 7f388bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41776/0 (socket says 172.21.15.76:41776) 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.841+0000 7f388bfff700 1 -- 172.21.15.76:0/1150360553 learned_addr learned my addr 172.21.15.76:0/1150360553 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:27.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.841+0000 7f388bfff700 1 -- 172.21.15.76:0/1150360553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f387c009d70 con 0x7f388c0f6590 2023-12-26T21:59:27.843 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.842+0000 7f388bfff700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c1067c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f387c013f10 tx=0x7f387c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:27.843 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.842+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387c00e040 con 0x7f388c0f6590 2023-12-26T21:59:27.843 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.842+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f387c007df0 con 0x7f388c0f6590 2023-12-26T21:59:27.843 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.842+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f387c014d60 con 0x7f388c0f6590 2023-12-26T21:59:27.843 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.842+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f388c106eb0 con 0x7f388c0f6590 2023-12-26T21:59:27.843 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.842+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f388c107340 con 0x7f388c0f6590 2023-12-26T21:59:27.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.843+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f387c01f460 con 0x7f388c0f6590 2023-12-26T21:59:27.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.843+0000 7f3888ff9700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3874031090 0x7f3874033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:27.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.843+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f387c042550 con 0x7f388c0f6590 2023-12-26T21:59:27.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.843+0000 7f388b7fe700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3874031090 0x7f3874033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:27.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.843+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f388c0591e0 con 0x7f388c0f6590 2023-12-26T21:59:27.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.843+0000 7f388b7fe700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3874031090 0x7f3874033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f38800096a0 tx=0x7f3880006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:27.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:27.848+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f387c0283f0 con 0x7f388c0f6590 2023-12-26T21:59:28.056 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.056+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f388c045bb0 con 0x7f388c0f6590 2023-12-26T21:59:28.057 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.057+0000 7f3888ff9700 1 -- 172.21.15.76:0/1150360553 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f387c012330 con 0x7f388c0f6590 2023-12-26T21:59:28.057 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:28.057 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3874031090 msgr2=0x7f3874033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3874031090 0x7f3874033550 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f38800096a0 tx=0x7f3880006b40).stop 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 msgr2=0x7f388c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c1067c0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f387c013f10 tx=0x7f387c009ad0).stop 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 shutdown_connections 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3874031090 0x7f3874033550 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 --2- 172.21.15.76:0/1150360553 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f388c0f6590 0x7f388c1067c0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 >> 172.21.15.76:0/1150360553 conn(0x7f388c0f1cd0 msgr2=0x7f388c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:28.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 shutdown_connections 2023-12-26T21:59:28.060 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:28.058+0000 7f3892812700 1 -- 172.21.15.76:0/1150360553 wait complete. 2023-12-26T21:59:28.060 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:27.348311+0000 mgr.smithi059.gwkiyc (mgr.14156) 83 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:28.058483+0000 mon.smithi059 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.76:0/1150360553' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:29.428 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:29.428 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:30.676 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 -- 172.21.15.76:0/872894533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 msgr2=0x7f14680f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:30.676 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 --2- 172.21.15.76:0/872894533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f14680f56f0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f1464009fb0 tx=0x7f1464009b30).stop 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 -- 172.21.15.76:0/872894533 shutdown_connections 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 --2- 172.21.15.76:0/872894533 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f14680f56f0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 -- 172.21.15.76:0/872894533 >> 172.21.15.76:0/872894533 conn(0x7f14680f0a30 msgr2=0x7f14680f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 -- 172.21.15.76:0/872894533 shutdown_connections 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 -- 172.21.15.76:0/872894533 wait complete. 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 Processor -- start 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.676+0000 7f146e588700 1 -- start start 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146e588700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f1468109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146e588700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1468060300 con 0x7f14680f52d0 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146d586700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f1468109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146d586700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f1468109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41782/0 (socket says 172.21.15.76:41782) 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146d586700 1 -- 172.21.15.76:0/2874487223 learned_addr learned my addr 172.21.15.76:0/2874487223 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146d586700 1 -- 172.21.15.76:0/2874487223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464005040 con 0x7f14680f52d0 2023-12-26T21:59:30.677 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.677+0000 7f146d586700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f1468109330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f14640089f0 tx=0x7f146400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.678+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1464009320 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.678+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1464013ec0 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.678+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1468060500 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.678+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1464014d30 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.678+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14681099e0 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.679+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f14640095f0 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.679+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14680591e0 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.679+0000 7f145dffb700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1454031090 0x7f1454033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.679+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f14640423e0 con 0x7f14680f52d0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.679+0000 7f146cd85700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1454031090 0x7f1454033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:30.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.679+0000 7f146cd85700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1454031090 0x7f1454033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f14580096a0 tx=0x7f1458006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:30.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.684+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f146401d7f0 con 0x7f14680f52d0 2023-12-26T21:59:30.889 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.890+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1468045bb0 con 0x7f14680f52d0 2023-12-26T21:59:30.890 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.890+0000 7f145dffb700 1 -- 172.21.15.76:0/2874487223 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f146401d7f0 con 0x7f14680f52d0 2023-12-26T21:59:30.890 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:30.890 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1454031090 msgr2=0x7f1454033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1454031090 0x7f1454033550 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f14580096a0 tx=0x7f1458006b40).stop 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 msgr2=0x7f1468109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f1468109330 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f14640089f0 tx=0x7f146400b040).stop 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 shutdown_connections 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1454031090 0x7f1454033550 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 --2- 172.21.15.76:0/2874487223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f14680f52d0 0x7f1468109330 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 >> 172.21.15.76:0/2874487223 conn(0x7f14680f0a30 msgr2=0x7f14680f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 shutdown_connections 2023-12-26T21:59:30.892 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:30.892+0000 7f146e588700 1 -- 172.21.15.76:0/2874487223 wait complete. 2023-12-26T21:59:30.893 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:29.348678+0000 mgr.smithi059.gwkiyc (mgr.14156) 84 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:30.891057+0000 mon.smithi059 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.76:0/2874487223' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:32.293 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:32.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:31.348952+0000 mgr.smithi059.gwkiyc (mgr.14156) 85 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:33.498 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 -- 172.21.15.76:0/712425844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 msgr2=0x7f9eec0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 --2- 172.21.15.76:0/712425844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec0f69b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f9ee0009fb0 tx=0x7f9ee0009b30).stop 2023-12-26T21:59:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 -- 172.21.15.76:0/712425844 shutdown_connections 2023-12-26T21:59:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 --2- 172.21.15.76:0/712425844 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec0f69b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 -- 172.21.15.76:0/712425844 >> 172.21.15.76:0/712425844 conn(0x7f9eec0f1d10 msgr2=0x7f9eec0f4130 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 -- 172.21.15.76:0/712425844 shutdown_connections 2023-12-26T21:59:33.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.499+0000 7f9ef2b70700 1 -- 172.21.15.76:0/712425844 wait complete. 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.500+0000 7f9ef2b70700 1 Processor -- start 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.500+0000 7f9ef2b70700 1 -- start start 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.500+0000 7f9ef2b70700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec1067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.500+0000 7f9ef2b70700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9eec106cd0 con 0x7f9eec0f6590 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.500+0000 7f9ef090c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec1067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ef090c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec1067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41794/0 (socket says 172.21.15.76:41794) 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ef090c700 1 -- 172.21.15.76:0/1278269417 learned_addr learned my addr 172.21.15.76:0/1278269417 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ef090c700 1 -- 172.21.15.76:0/1278269417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ee0005040 con 0x7f9eec0f6590 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ef090c700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec1067e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f9ee00089f0 tx=0x7f9ee000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee0009320 con 0x7f9eec0f6590 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9eec106ed0 con 0x7f9eec0f6590 2023-12-26T21:59:33.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9ee0013ec0 con 0x7f9eec0f6590 2023-12-26T21:59:33.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ee0014d30 con 0x7f9eec0f6590 2023-12-26T21:59:33.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.501+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9eec107360 con 0x7f9eec0f6590 2023-12-26T21:59:33.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.502+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9ee00095f0 con 0x7f9eec0f6590 2023-12-26T21:59:33.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.502+0000 7f9ee97fa700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9edc031090 0x7f9edc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:33.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.502+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9ee00424e0 con 0x7f9eec0f6590 2023-12-26T21:59:33.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.502+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9eec045bb0 con 0x7f9eec0f6590 2023-12-26T21:59:33.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.502+0000 7f9eebfff700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9edc031090 0x7f9edc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:33.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.503+0000 7f9eebfff700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9edc031090 0x7f9edc033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9ed80096a0 tx=0x7f9ed8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:33.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.507+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9ee001d7f0 con 0x7f9eec0f6590 2023-12-26T21:59:33.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.716+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9eec0fc170 con 0x7f9eec0f6590 2023-12-26T21:59:33.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.717+0000 7f9ee97fa700 1 -- 172.21.15.76:0/1278269417 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9ee002b020 con 0x7f9eec0f6590 2023-12-26T21:59:33.716 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:33.717 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9edc031090 msgr2=0x7f9edc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9edc031090 0x7f9edc033550 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f9ed80096a0 tx=0x7f9ed8006b40).stop 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 msgr2=0x7f9eec1067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec1067e0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f9ee00089f0 tx=0x7f9ee000b040).stop 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 shutdown_connections 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9edc031090 0x7f9edc033550 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 --2- 172.21.15.76:0/1278269417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9eec0f6590 0x7f9eec1067e0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 >> 172.21.15.76:0/1278269417 conn(0x7f9eec0f1d10 msgr2=0x7f9eec0f2800 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 shutdown_connections 2023-12-26T21:59:33.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:33.718+0000 7f9ef2b70700 1 -- 172.21.15.76:0/1278269417 wait complete. 2023-12-26T21:59:33.719 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:34.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:33 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:33.717485+0000 mon.smithi059 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.76:0/1278269417' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:35.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:35.096 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:33.349208+0000 mgr.smithi059.gwkiyc (mgr.14156) 86 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:36.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 -- 172.21.15.76:0/814279153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 msgr2=0x7fe8740f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 --2- 172.21.15.76:0/814279153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe8740f56f0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe864009fb0 tx=0x7fe864009b30).stop 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 -- 172.21.15.76:0/814279153 shutdown_connections 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 --2- 172.21.15.76:0/814279153 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe8740f56f0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 -- 172.21.15.76:0/814279153 >> 172.21.15.76:0/814279153 conn(0x7fe8740f0a30 msgr2=0x7fe8740f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 -- 172.21.15.76:0/814279153 shutdown_connections 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.319+0000 7fe87b4a1700 1 -- 172.21.15.76:0/814279153 wait complete. 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87b4a1700 1 Processor -- start 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87b4a1700 1 -- start start 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87b4a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe874105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87b4a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe874105a00 con 0x7fe8740f52d0 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87a49f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe874105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:36.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87a49f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe874105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47354/0 (socket says 172.21.15.76:47354) 2023-12-26T21:59:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87a49f700 1 -- 172.21.15.76:0/3156857875 learned_addr learned my addr 172.21.15.76:0/3156857875 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.320+0000 7fe87a49f700 1 -- 172.21.15.76:0/3156857875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe864005040 con 0x7fe8740f52d0 2023-12-26T21:59:36.321 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.321+0000 7fe87a49f700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe874105510 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fe8640089f0 tx=0x7fe86400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.321+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe864009320 con 0x7fe8740f52d0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.321+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe864013ec0 con 0x7fe8740f52d0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.321+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe874105c00 con 0x7fe8740f52d0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.321+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe864014d30 con 0x7fe8740f52d0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.321+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe874106090 con 0x7fe8740f52d0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.322+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe864014420 con 0x7fe8740f52d0 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.322+0000 7fe86affd700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe860031090 0x7fe860033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:36.322 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.322+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe874045bb0 con 0x7fe8740f52d0 2023-12-26T21:59:36.323 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.322+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe864042400 con 0x7fe8740f52d0 2023-12-26T21:59:36.323 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.322+0000 7fe879c9e700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe860031090 0x7fe860033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:36.323 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.323+0000 7fe879c9e700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe860031090 0x7fe860033550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe8700096a0 tx=0x7fe870006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:36.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.327+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe864014750 con 0x7fe8740f52d0 2023-12-26T21:59:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.533+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe874106e90 con 0x7fe8740f52d0 2023-12-26T21:59:36.533 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.534+0000 7fe86affd700 1 -- 172.21.15.76:0/3156857875 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe86401d380 con 0x7fe8740f52d0 2023-12-26T21:59:36.533 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:36.534 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe860031090 msgr2=0x7fe860033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe860031090 0x7fe860033550 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe8700096a0 tx=0x7fe870006b40).stop 2023-12-26T21:59:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 msgr2=0x7fe874105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:36.535 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe874105510 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fe8640089f0 tx=0x7fe86400b040).stop 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 shutdown_connections 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe860031090 0x7fe860033550 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 --2- 172.21.15.76:0/3156857875 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8740f52d0 0x7fe874105510 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 >> 172.21.15.76:0/3156857875 conn(0x7fe8740f0a30 msgr2=0x7fe8740f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 shutdown_connections 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:36.536+0000 7fe87b4a1700 1 -- 172.21.15.76:0/3156857875 wait complete. 2023-12-26T21:59:36.536 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:35.349498+0000 mgr.smithi059.gwkiyc (mgr.14156) 87 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:36.534507+0000 mon.smithi059 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.76:0/3156857875' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:37.923 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:37.924 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.135+0000 7f54c5264700 1 -- 172.21.15.76:0/1733917634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 msgr2=0x7f54c00f5660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.135+0000 7f54c5264700 1 --2- 172.21.15.76:0/1733917634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c00f5660 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f54b0009fb0 tx=0x7f54b0009b30).stop 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.135+0000 7f54c5264700 1 -- 172.21.15.76:0/1733917634 shutdown_connections 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.135+0000 7f54c5264700 1 --2- 172.21.15.76:0/1733917634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c00f5660 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.135+0000 7f54c5264700 1 -- 172.21.15.76:0/1733917634 >> 172.21.15.76:0/1733917634 conn(0x7f54c00f09a0 msgr2=0x7f54c00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.136+0000 7f54c5264700 1 -- 172.21.15.76:0/1733917634 shutdown_connections 2023-12-26T21:59:39.135 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.136+0000 7f54c5264700 1 -- 172.21.15.76:0/1733917634 wait complete. 2023-12-26T21:59:39.136 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.136+0000 7f54c5264700 1 Processor -- start 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.136+0000 7f54c5264700 1 -- start start 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.136+0000 7f54c5264700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c01065c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.136+0000 7f54c5264700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c0106ab0 con 0x7f54c00f5240 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c01065c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c01065c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47386/0 (socket says 172.21.15.76:47386) 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bffff700 1 -- 172.21.15.76:0/496935744 learned_addr learned my addr 172.21.15.76:0/496935744 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bffff700 1 -- 172.21.15.76:0/496935744 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b0005040 con 0x7f54c00f5240 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bffff700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c01065c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f54b00089f0 tx=0x7f54b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b0009320 con 0x7f54c00f5240 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f54b0013ec0 con 0x7f54c00f5240 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c0106cb0 con 0x7f54c00f5240 2023-12-26T21:59:39.137 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54b0014d30 con 0x7f54c00f5240 2023-12-26T21:59:39.138 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.137+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c0107140 con 0x7f54c00f5240 2023-12-26T21:59:39.138 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.138+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f54b00095f0 con 0x7f54c00f5240 2023-12-26T21:59:39.138 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.138+0000 7f54bcff9700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54a8031090 0x7f54a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:39.138 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.138+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c0045bb0 con 0x7f54c00f5240 2023-12-26T21:59:39.139 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.138+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f54b00424a0 con 0x7f54c00f5240 2023-12-26T21:59:39.139 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.139+0000 7f54bf7fe700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54a8031090 0x7f54a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:39.139 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.139+0000 7f54bf7fe700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54a8031090 0x7f54a8033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f54b40096a0 tx=0x7f54b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:39.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.143+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f54b000fe40 con 0x7f54c00f5240 2023-12-26T21:59:39.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:37.349780+0000 mgr.smithi059.gwkiyc (mgr.14156) 88 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:39.351 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.351+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54c0107ff0 con 0x7f54c00f5240 2023-12-26T21:59:39.351 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.352+0000 7f54bcff9700 1 -- 172.21.15.76:0/496935744 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54b000fe40 con 0x7f54c00f5240 2023-12-26T21:59:39.351 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:39.351 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54a8031090 msgr2=0x7f54a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54a8031090 0x7f54a8033550 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f54b40096a0 tx=0x7f54b4006b40).stop 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 msgr2=0x7f54c01065c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c01065c0 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f54b00089f0 tx=0x7f54b000b040).stop 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 shutdown_connections 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54a8031090 0x7f54a8033550 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.353+0000 7f54c5264700 1 --2- 172.21.15.76:0/496935744 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54c00f5240 0x7f54c01065c0 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.354+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 >> 172.21.15.76:0/496935744 conn(0x7f54c00f09a0 msgr2=0x7f54c00f1500 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.354+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 shutdown_connections 2023-12-26T21:59:39.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:39.354+0000 7f54c5264700 1 -- 172.21.15.76:0/496935744 wait complete. 2023-12-26T21:59:39.354 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:40.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:39 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:39.352667+0000 mon.smithi059 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.76:0/496935744' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:40.737 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:40.737 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:39.350129+0000 mgr.smithi059.gwkiyc (mgr.14156) 89 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 -- 172.21.15.76:0/2308668736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 msgr2=0x7fa2e40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 --2- 172.21.15.76:0/2308668736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e40f58a0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fa2d8009fb0 tx=0x7fa2d8009b30).stop 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 -- 172.21.15.76:0/2308668736 shutdown_connections 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 --2- 172.21.15.76:0/2308668736 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e40f58a0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 -- 172.21.15.76:0/2308668736 >> 172.21.15.76:0/2308668736 conn(0x7fa2e40f0be0 msgr2=0x7fa2e40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 -- 172.21.15.76:0/2308668736 shutdown_connections 2023-12-26T21:59:41.926 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.926+0000 7fa2eab95700 1 -- 172.21.15.76:0/2308668736 wait complete. 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.927+0000 7fa2eab95700 1 Processor -- start 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.927+0000 7fa2eab95700 1 -- start start 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.927+0000 7fa2eab95700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e41094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.927+0000 7fa2eab95700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2e4060360 con 0x7fa2e40f5480 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e8931700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e41094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e8931700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e41094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47408/0 (socket says 172.21.15.76:47408) 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e8931700 1 -- 172.21.15.76:0/1134742385 learned_addr learned my addr 172.21.15.76:0/1134742385 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e8931700 1 -- 172.21.15.76:0/1134742385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2d8005040 con 0x7fa2e40f5480 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e8931700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e41094b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fa2d80089f0 tx=0x7fa2d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2d8009320 con 0x7fa2e40f5480 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2e4060560 con 0x7fa2e40f5480 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa2d8013ec0 con 0x7fa2e40f5480 2023-12-26T21:59:41.928 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2d8014d30 con 0x7fa2e40f5480 2023-12-26T21:59:41.929 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.928+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2e4109be0 con 0x7fa2e40f5480 2023-12-26T21:59:41.929 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.929+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa2d80095f0 con 0x7fa2e40f5480 2023-12-26T21:59:41.929 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.929+0000 7fa2e17fa700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2d4031090 0x7fa2d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:41.930 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.929+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2e4045bb0 con 0x7fa2e40f5480 2023-12-26T21:59:41.930 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.929+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2d80423e0 con 0x7fa2e40f5480 2023-12-26T21:59:41.930 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.929+0000 7fa2e3fff700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2d4031090 0x7fa2d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:41.930 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.930+0000 7fa2e3fff700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2d4031090 0x7fa2d4033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa2d00096a0 tx=0x7fa2d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:41.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:41.934+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2d801dd70 con 0x7fa2e40f5480 2023-12-26T21:59:42.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.146+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2e40faf70 con 0x7fa2e40f5480 2023-12-26T21:59:42.145 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.146+0000 7fa2e17fa700 1 -- 172.21.15.76:0/1134742385 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2d801d380 con 0x7fa2e40f5480 2023-12-26T21:59:42.146 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:42.146 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:42.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2d4031090 msgr2=0x7fa2d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:42.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2d4031090 0x7fa2d4033550 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa2d00096a0 tx=0x7fa2d0006b40).stop 2023-12-26T21:59:42.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 msgr2=0x7fa2e41094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e41094b0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fa2d80089f0 tx=0x7fa2d800b040).stop 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 shutdown_connections 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2d4031090 0x7fa2d4033550 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 --2- 172.21.15.76:0/1134742385 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2e40f5480 0x7fa2e41094b0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 >> 172.21.15.76:0/1134742385 conn(0x7fa2e40f0be0 msgr2=0x7fa2e40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 shutdown_connections 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:42.148+0000 7fa2eab95700 1 -- 172.21.15.76:0/1134742385 wait complete. 2023-12-26T21:59:42.148 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:41.350500+0000 mgr.smithi059.gwkiyc (mgr.14156) 90 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:42.146790+0000 mon.smithi059 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.76:0/1134742385' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:43.494 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:43.495 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.757+0000 7f9ea7396700 1 -- 172.21.15.76:0/2283606272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 msgr2=0x7f9ea00f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.757+0000 7f9ea7396700 1 --2- 172.21.15.76:0/2283606272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea00f69b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f9e90004760 tx=0x7f9e90009ad0).stop 2023-12-26T21:59:44.758 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.757+0000 7f9ea7396700 1 -- 172.21.15.76:0/2283606272 shutdown_connections 2023-12-26T21:59:44.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.757+0000 7f9ea7396700 1 --2- 172.21.15.76:0/2283606272 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea00f69b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:44.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.757+0000 7f9ea7396700 1 -- 172.21.15.76:0/2283606272 >> 172.21.15.76:0/2283606272 conn(0x7f9ea00f1cd0 msgr2=0x7f9ea00f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:44.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.757+0000 7f9ea7396700 1 -- 172.21.15.76:0/2283606272 shutdown_connections 2023-12-26T21:59:44.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea7396700 1 -- 172.21.15.76:0/2283606272 wait complete. 2023-12-26T21:59:44.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea7396700 1 Processor -- start 2023-12-26T21:59:44.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea7396700 1 -- start start 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea7396700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea01067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea7396700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea0106cb0 con 0x7f9ea00f6590 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea5132700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea01067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea5132700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea01067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47412/0 (socket says 172.21.15.76:47412) 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.758+0000 7f9ea5132700 1 -- 172.21.15.76:0/784058016 learned_addr learned my addr 172.21.15.76:0/784058016 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9ea5132700 1 -- 172.21.15.76:0/784058016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e90009d70 con 0x7f9ea00f6590 2023-12-26T21:59:44.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9ea5132700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea01067c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9e90004170 tx=0x7f9e90009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9000e040 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea0106eb0 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9e90008ef0 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e90014c90 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.759+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea0107340 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.760+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9e90014420 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.760+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea0045bb0 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.760+0000 7f9e95ffb700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e8c0310e0 0x7f9e8c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.760+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9e90042300 con 0x7f9ea00f6590 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.760+0000 7f9ea4931700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e8c0310e0 0x7f9e8c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:44.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.761+0000 7f9ea4931700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e8c0310e0 0x7f9e8c0335a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f9e9c0096a0 tx=0x7f9e9c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:44.765 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.765+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9e90026300 con 0x7f9ea00f6590 2023-12-26T21:59:44.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.970+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ea01081f0 con 0x7f9ea00f6590 2023-12-26T21:59:44.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.971+0000 7f9e95ffb700 1 -- 172.21.15.76:0/784058016 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e90012b20 con 0x7f9ea00f6590 2023-12-26T21:59:44.971 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:44.971 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e8c0310e0 msgr2=0x7f9e8c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e8c0310e0 0x7f9e8c0335a0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f9e9c0096a0 tx=0x7f9e9c006b40).stop 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 msgr2=0x7f9ea01067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea01067c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f9e90004170 tx=0x7f9e90009ad0).stop 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 shutdown_connections 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e8c0310e0 0x7f9e8c0335a0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 --2- 172.21.15.76:0/784058016 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9ea00f6590 0x7f9ea01067c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 >> 172.21.15.76:0/784058016 conn(0x7f9ea00f1cd0 msgr2=0x7f9ea00f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 shutdown_connections 2023-12-26T21:59:44.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:44.972+0000 7f9ea7396700 1 -- 172.21.15.76:0/784058016 wait complete. 2023-12-26T21:59:44.974 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:45.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:43.350737+0000 mgr.smithi059.gwkiyc (mgr.14156) 91 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:44.972245+0000 mon.smithi059 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.76:0/784058016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:45.467317+0000 mon.smithi059 (mon.0) 167 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T21:59:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:45.467754+0000 mon.smithi059 (mon.0) 168 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T21:59:46.369 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:46.369 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:47.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:45.351031+0000 mgr.smithi059.gwkiyc (mgr.14156) 92 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:47.584 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.582+0000 7fe9ef21f700 1 -- 172.21.15.76:0/750884076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 msgr2=0x7fe9e80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:47.584 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.582+0000 7fe9ef21f700 1 --2- 172.21.15.76:0/750884076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e80f56f0 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe9d8009fb0 tx=0x7fe9d8009b30).stop 2023-12-26T21:59:47.584 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.582+0000 7fe9ef21f700 1 -- 172.21.15.76:0/750884076 shutdown_connections 2023-12-26T21:59:47.584 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.582+0000 7fe9ef21f700 1 --2- 172.21.15.76:0/750884076 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e80f56f0 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 -- 172.21.15.76:0/750884076 >> 172.21.15.76:0/750884076 conn(0x7fe9e80f0a30 msgr2=0x7fe9e80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 -- 172.21.15.76:0/750884076 shutdown_connections 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 -- 172.21.15.76:0/750884076 wait complete. 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 Processor -- start 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 -- start start 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e810b730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ef21f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e8107120 con 0x7fe9e80f52d0 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ee21d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e810b730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ee21d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e810b730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40284/0 (socket says 172.21.15.76:40284) 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.583+0000 7fe9ee21d700 1 -- 172.21.15.76:0/265156535 learned_addr learned my addr 172.21.15.76:0/265156535 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9ee21d700 1 -- 172.21.15.76:0/265156535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d8005040 con 0x7fe9e80f52d0 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9ee21d700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e810b730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe9d8000c00 tx=0x7fe9d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:47.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d80095f0 con 0x7fe9e80f52d0 2023-12-26T21:59:47.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e8060300 con 0x7fe9e80f52d0 2023-12-26T21:59:47.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe9d8007e50 con 0x7fe9e80f52d0 2023-12-26T21:59:47.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9d8014e70 con 0x7fe9e80f52d0 2023-12-26T21:59:47.586 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.584+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e810bc20 con 0x7fe9e80f52d0 2023-12-26T21:59:47.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.585+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe9d8025070 con 0x7fe9e80f52d0 2023-12-26T21:59:47.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.585+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e8045bb0 con 0x7fe9e80f52d0 2023-12-26T21:59:47.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.585+0000 7fe9deffd700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe9d4031090 0x7fe9d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:47.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.585+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe9d8041d40 con 0x7fe9e80f52d0 2023-12-26T21:59:47.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.585+0000 7fe9eda1c700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe9d4031090 0x7fe9d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:47.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.586+0000 7fe9eda1c700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe9d4031090 0x7fe9d4033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe9e40096a0 tx=0x7fe9e4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:47.590 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.590+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe9d800f300 con 0x7fe9e80f52d0 2023-12-26T21:59:47.798 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.797+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9e810c3d0 con 0x7fe9e80f52d0 2023-12-26T21:59:47.798 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.798+0000 7fe9deffd700 1 -- 172.21.15.76:0/265156535 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe9d800f300 con 0x7fe9e80f52d0 2023-12-26T21:59:47.798 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:47.799 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe9d4031090 msgr2=0x7fe9d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe9d4031090 0x7fe9d4033550 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fe9e40096a0 tx=0x7fe9e4006b40).stop 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 msgr2=0x7fe9e810b730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e810b730 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fe9d8000c00 tx=0x7fe9d800b040).stop 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 shutdown_connections 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe9d4031090 0x7fe9d4033550 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 --2- 172.21.15.76:0/265156535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe9e80f52d0 0x7fe9e810b730 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:47.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 >> 172.21.15.76:0/265156535 conn(0x7fe9e80f0a30 msgr2=0x7fe9e80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:47.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 shutdown_connections 2023-12-26T21:59:47.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:47.800+0000 7fe9ef21f700 1 -- 172.21.15.76:0/265156535 wait complete. 2023-12-26T21:59:47.801 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:48.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:47 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:47.799726+0000 mon.smithi059 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.76:0/265156535' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:49.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:49.185 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:47.351375+0000 mgr.smithi059.gwkiyc (mgr.14156) 93 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.387+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/575868995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 msgr2=0x7f8ce40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:50.391 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.387+0000 7f8ce9ebc700 1 --2- 172.21.15.76:0/575868995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce40f56f0 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f8cd8009fb0 tx=0x7f8cd8009b30).stop 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/575868995 shutdown_connections 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 --2- 172.21.15.76:0/575868995 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce40f56f0 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/575868995 >> 172.21.15.76:0/575868995 conn(0x7f8ce40f0a30 msgr2=0x7f8ce40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/575868995 shutdown_connections 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/575868995 wait complete. 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 Processor -- start 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 -- start start 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce4106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.388+0000 7f8ce9ebc700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ce4106b10 con 0x7f8ce40f52d0 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce8eba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce4106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce8eba700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce4106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40302/0 (socket says 172.21.15.76:40302) 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce8eba700 1 -- 172.21.15.76:0/2579068169 learned_addr learned my addr 172.21.15.76:0/2579068169 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:50.392 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce8eba700 1 -- 172.21.15.76:0/2579068169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd8005040 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce8eba700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce4106620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8cd80089f0 tx=0x7f8cd800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd8009320 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8cd8013ec0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cd8014d30 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ce4106d10 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.389+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ce41071a0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.390+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8cd80095f0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.390+0000 7f8ce17fa700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cd4031090 0x7f8cd4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:50.393 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.390+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8cd80423e0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.394 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.390+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ce4045bb0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.394 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.390+0000 7f8ce3fff700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cd4031090 0x7f8cd4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:50.394 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.391+0000 7f8ce3fff700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cd4031090 0x7f8cd4033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8cd00096a0 tx=0x7f8cd0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:50.395 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.395+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f8cd801d7f0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.604+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ce4107ef0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.605 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.605+0000 7f8ce17fa700 1 -- 172.21.15.76:0/2579068169 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8cd801d7f0 con 0x7f8ce40f52d0 2023-12-26T21:59:50.606 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:50.606 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:50.607 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cd4031090 msgr2=0x7f8cd4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:50.607 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cd4031090 0x7f8cd4033550 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f8cd00096a0 tx=0x7f8cd0006b40).stop 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 msgr2=0x7f8ce4106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce4106620 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f8cd80089f0 tx=0x7f8cd800b040).stop 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 shutdown_connections 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8cd4031090 0x7f8cd4033550 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 --2- 172.21.15.76:0/2579068169 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f8ce40f52d0 0x7f8ce4106620 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 >> 172.21.15.76:0/2579068169 conn(0x7f8ce40f0a30 msgr2=0x7f8ce405ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 shutdown_connections 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:50.607+0000 7f8ce9ebc700 1 -- 172.21.15.76:0/2579068169 wait complete. 2023-12-26T21:59:50.608 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:49.351665+0000 mgr.smithi059.gwkiyc (mgr.14156) 94 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:50.606567+0000 mon.smithi059 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.76:0/2579068169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:51.993 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:51.996 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:53.238 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 -- 172.21.15.76:0/1098371784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 msgr2=0x7fefc00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:53.238 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 --2- 172.21.15.76:0/1098371784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f9160 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fefb4009fb0 tx=0x7fefb4009b30).stop 2023-12-26T21:59:53.238 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 -- 172.21.15.76:0/1098371784 shutdown_connections 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 --2- 172.21.15.76:0/1098371784 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f9160 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 -- 172.21.15.76:0/1098371784 >> 172.21.15.76:0/1098371784 conn(0x7fefc00f0be0 msgr2=0x7fefc00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 -- 172.21.15.76:0/1098371784 shutdown_connections 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 -- 172.21.15.76:0/1098371784 wait complete. 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 Processor -- start 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.237+0000 7fefc6df9700 1 -- start start 2023-12-26T21:59:53.239 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc6df9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:53.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc6df9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fefc00f5060 con 0x7fefc0060360 2023-12-26T21:59:53.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc4b95700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:53.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc4b95700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:40316/0 (socket says 172.21.15.76:40316) 2023-12-26T21:59:53.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc4b95700 1 -- 172.21.15.76:0/393419482 learned_addr learned my addr 172.21.15.76:0/393419482 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:53.240 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc4b95700 1 -- 172.21.15.76:0/393419482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefb4005040 con 0x7fefc0060360 2023-12-26T21:59:53.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefc4b95700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f6be0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fefb4009f80 tx=0x7fefb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:53.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.238+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefb4009320 con 0x7fefc0060360 2023-12-26T21:59:53.241 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fefc00f5200 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fefb4013680 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fefb401eda0 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fefc00f5690 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fefb401e420 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefbd7fa700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fefb0031090 0x7fefb0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fefb4042470 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.239+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fefc00591e0 con 0x7fefc0060360 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.240+0000 7fefbffff700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fefb0031090 0x7fefb0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:53.242 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.240+0000 7fefbffff700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fefb0031090 0x7fefb0033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fefac0096a0 tx=0x7fefac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:53.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.244+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fefb401d380 con 0x7fefc0060360 2023-12-26T21:59:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:51.352002+0000 mgr.smithi059.gwkiyc (mgr.14156) 95 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:53.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.448+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fefc00f5df0 con 0x7fefc0060360 2023-12-26T21:59:53.449 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.448+0000 7fefbd7fa700 1 -- 172.21.15.76:0/393419482 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fefb401cb10 con 0x7fefc0060360 2023-12-26T21:59:53.449 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:53.449 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:53.450 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fefb0031090 msgr2=0x7fefb0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fefb0031090 0x7fefb0033550 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fefac0096a0 tx=0x7fefac006b40).stop 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 msgr2=0x7fefc00f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f6be0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fefb4009f80 tx=0x7fefb400b040).stop 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 shutdown_connections 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fefb0031090 0x7fefb0033550 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 --2- 172.21.15.76:0/393419482 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fefc0060360 0x7fefc00f6be0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 >> 172.21.15.76:0/393419482 conn(0x7fefc00f0be0 msgr2=0x7fefc00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 shutdown_connections 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:53.450+0000 7fefc6df9700 1 -- 172.21.15.76:0/393419482 wait complete. 2023-12-26T21:59:53.451 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:54.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:53 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:53.449935+0000 mon.smithi059 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.76:0/393419482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:54.847 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:54.848 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:53.352395+0000 mgr.smithi059.gwkiyc (mgr.14156) 96 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 -- 172.21.15.76:0/2015881550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 msgr2=0x7f77940f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 --2- 172.21.15.76:0/2015881550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77940f58a0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f777c009fb0 tx=0x7f777c009b30).stop 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 -- 172.21.15.76:0/2015881550 shutdown_connections 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 --2- 172.21.15.76:0/2015881550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77940f58a0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 -- 172.21.15.76:0/2015881550 >> 172.21.15.76:0/2015881550 conn(0x7f77940f0be0 msgr2=0x7f77940f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 -- 172.21.15.76:0/2015881550 shutdown_connections 2023-12-26T21:59:56.059 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.058+0000 7f77992a2700 1 -- 172.21.15.76:0/2015881550 wait complete. 2023-12-26T21:59:56.061 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f77992a2700 1 Processor -- start 2023-12-26T21:59:56.061 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f77992a2700 1 -- start start 2023-12-26T21:59:56.061 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f77992a2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77941056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:56.061 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f77992a2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7794105be0 con 0x7f77940f5480 2023-12-26T21:59:56.061 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f7792ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77941056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f7792ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77941056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57872/0 (socket says 172.21.15.76:57872) 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.059+0000 7f7792ffd700 1 -- 172.21.15.76:0/3400674231 learned_addr learned my addr 172.21.15.76:0/3400674231 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f7792ffd700 1 -- 172.21.15.76:0/3400674231 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f777c005040 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f7792ffd700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77941056f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f777c0089f0 tx=0x7f777c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777c009320 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7794105de0 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f777c013ec0 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f777c014d30 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.060+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7794106270 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.061+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f777c0095f0 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.061+0000 7f778b7fe700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7780031090 0x7f7780033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.061+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7794045bb0 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.061+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f777c0424a0 con 0x7f77940f5480 2023-12-26T21:59:56.062 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.061+0000 7f77927fc700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7780031090 0x7f7780033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:56.063 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.062+0000 7f77927fc700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7780031090 0x7f7780033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f77840096a0 tx=0x7f7784006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:56.066 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.066+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f777c01d980 con 0x7f77940f5480 2023-12-26T21:59:56.274 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.273+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77940fb060 con 0x7f77940f5480 2023-12-26T21:59:56.274 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.274+0000 7f778b7fe700 1 -- 172.21.15.76:0/3400674231 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f777c01d380 con 0x7f77940f5480 2023-12-26T21:59:56.274 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:56.274 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7780031090 msgr2=0x7f7780033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7780031090 0x7f7780033550 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f77840096a0 tx=0x7f7784006b40).stop 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 msgr2=0x7f77941056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77941056f0 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f777c0089f0 tx=0x7f777c00b040).stop 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 shutdown_connections 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7780031090 0x7f7780033550 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 --2- 172.21.15.76:0/3400674231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f77940f5480 0x7f77941056f0 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 >> 172.21.15.76:0/3400674231 conn(0x7f77940f0be0 msgr2=0x7f77940f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 shutdown_connections 2023-12-26T21:59:56.276 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:56.275+0000 7f77992a2700 1 -- 172.21.15.76:0/3400674231 wait complete. 2023-12-26T21:59:56.277 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:55.352687+0000 mgr.smithi059.gwkiyc (mgr.14156) 97 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T21:59:57.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:56.275411+0000 mon.smithi059 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.76:0/3400674231' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T21:59:57.668 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T21:59:57.669 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 -- 172.21.15.76:0/2936139222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 msgr2=0x7fc7600f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 --2- 172.21.15.76:0/2936139222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7600f58a0 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fc750009fb0 tx=0x7fc750009b30).stop 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 -- 172.21.15.76:0/2936139222 shutdown_connections 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 --2- 172.21.15.76:0/2936139222 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7600f58a0 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 -- 172.21.15.76:0/2936139222 >> 172.21.15.76:0/2936139222 conn(0x7fc7600f0be0 msgr2=0x7fc7600f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 -- 172.21.15.76:0/2936139222 shutdown_connections 2023-12-26T21:59:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.865+0000 7fc767cb4700 1 -- 172.21.15.76:0/2936139222 wait complete. 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc767cb4700 1 Processor -- start 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc767cb4700 1 -- start start 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc767cb4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7601067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc767cb4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc760106cd0 con 0x7fc7600f5480 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc765a50700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7601067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc765a50700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7601067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57890/0 (socket says 172.21.15.76:57890) 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc765a50700 1 -- 172.21.15.76:0/1249767257 learned_addr learned my addr 172.21.15.76:0/1249767257 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc765a50700 1 -- 172.21.15.76:0/1249767257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc750005040 con 0x7fc7600f5480 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.866+0000 7fc765a50700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7601067e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fc7500089f0 tx=0x7fc75000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.867+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc750009320 con 0x7fc7600f5480 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.867+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc760106ed0 con 0x7fc7600f5480 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.867+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc750013ec0 con 0x7fc7600f5480 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.867+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc750014d30 con 0x7fc7600f5480 2023-12-26T21:59:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.867+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc760107360 con 0x7fc7600f5480 2023-12-26T21:59:58.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.867+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc7500095f0 con 0x7fc7600f5480 2023-12-26T21:59:58.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.868+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc760045bb0 con 0x7fc7600f5480 2023-12-26T21:59:58.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.868+0000 7fc7567fc700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc74c031090 0x7fc74c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T21:59:58.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.868+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc7500424a0 con 0x7fc7600f5480 2023-12-26T21:59:58.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.868+0000 7fc76524f700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc74c031090 0x7fc74c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T21:59:58.870 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.868+0000 7fc76524f700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc74c031090 0x7fc74c033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fc75c0096a0 tx=0x7fc75c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T21:59:58.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:58.873+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc75001d7f0 con 0x7fc7600f5480 2023-12-26T21:59:59.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.082+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc760108210 con 0x7fc7600f5480 2023-12-26T21:59:59.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.083+0000 7fc7567fc700 1 -- 172.21.15.76:0/1249767257 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc75001d7f0 con 0x7fc7600f5480 2023-12-26T21:59:59.083 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T21:59:59.083 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc74c031090 msgr2=0x7fc74c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc74c031090 0x7fc74c033550 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fc75c0096a0 tx=0x7fc75c006b40).stop 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 msgr2=0x7fc7601067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7601067e0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7fc7500089f0 tx=0x7fc75000b040).stop 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 shutdown_connections 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc74c031090 0x7fc74c033550 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 --2- 172.21.15.76:0/1249767257 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc7600f5480 0x7fc7601067e0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 >> 172.21.15.76:0/1249767257 conn(0x7fc7600f0be0 msgr2=0x7fc76005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.084+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 shutdown_connections 2023-12-26T21:59:59.085 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T21:59:59.085+0000 7fc767cb4700 1 -- 172.21.15.76:0/1249767257 wait complete. 2023-12-26T21:59:59.086 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T21:59:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:57.353025+0000 mgr.smithi059.gwkiyc (mgr.14156) 98 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:00.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 21:59:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T21:59:59.084575+0000 mon.smithi059 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.76:0/1249767257' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:00.477 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:00.477 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T21:59:59.353473+0000 mgr.smithi059.gwkiyc (mgr.14156) 99 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.698+0000 7f2fb877f700 1 -- 172.21.15.76:0/2282436019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 msgr2=0x7f2fb00f5e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.698+0000 7f2fb877f700 1 --2- 172.21.15.76:0/2282436019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb00f5e80 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f2fa0009fb0 tx=0x7f2fa0009b30).stop 2023-12-26T22:00:01.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.698+0000 7f2fb877f700 1 -- 172.21.15.76:0/2282436019 shutdown_connections 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.698+0000 7f2fb877f700 1 --2- 172.21.15.76:0/2282436019 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb00f5e80 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.698+0000 7f2fb877f700 1 -- 172.21.15.76:0/2282436019 >> 172.21.15.76:0/2282436019 conn(0x7f2fb00f0390 msgr2=0x7f2fb00f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.698+0000 7f2fb877f700 1 -- 172.21.15.76:0/2282436019 shutdown_connections 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb877f700 1 -- 172.21.15.76:0/2282436019 wait complete. 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb877f700 1 Processor -- start 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb877f700 1 -- start start 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb877f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb0105700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb877f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fb0105bf0 con 0x7f2fb00f5a60 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb651b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb0105700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb651b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb0105700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57922/0 (socket says 172.21.15.76:57922) 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.699+0000 7f2fb651b700 1 -- 172.21.15.76:0/412239882 learned_addr learned my addr 172.21.15.76:0/412239882 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:01.701 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fb651b700 1 -- 172.21.15.76:0/412239882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fa0005040 con 0x7f2fb00f5a60 2023-12-26T22:00:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fb651b700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb0105700 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2fa0000c00 tx=0x7f2fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa00095f0 con 0x7f2fb00f5a60 2023-12-26T22:00:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fb0105df0 con 0x7f2fb00f5a60 2023-12-26T22:00:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2fa0007e50 con 0x7f2fb00f5a60 2023-12-26T22:00:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fa0014e70 con 0x7f2fb00f5a60 2023-12-26T22:00:01.702 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.700+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fb0106280 con 0x7f2fb00f5a60 2023-12-26T22:00:01.703 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.701+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2fa0025070 con 0x7f2fb00f5a60 2023-12-26T22:00:01.703 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.701+0000 7f2fa6ffd700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f9c0310e0 0x7f2f9c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.701+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2fa0042d70 con 0x7f2fb00f5a60 2023-12-26T22:00:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.701+0000 7f2fb5d1a700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f9c0310e0 0x7f2f9c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.701+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fb00591e0 con 0x7f2fb00f5a60 2023-12-26T22:00:01.704 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.702+0000 7f2fb5d1a700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f9c0310e0 0x7f2f9c0335a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2fac0096a0 tx=0x7f2fac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:01.706 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.706+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2fa0026350 con 0x7f2fb00f5a60 2023-12-26T22:00:01.911 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.910+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fb0045bb0 con 0x7f2fb00f5a60 2023-12-26T22:00:01.911 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.911+0000 7f2fa6ffd700 1 -- 172.21.15.76:0/412239882 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2fa0012330 con 0x7f2fb00f5a60 2023-12-26T22:00:01.911 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:01.911 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:01.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f9c0310e0 msgr2=0x7f2f9c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:01.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f9c0310e0 0x7f2f9c0335a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2fac0096a0 tx=0x7f2fac006b40).stop 2023-12-26T22:00:01.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 msgr2=0x7f2fb0105700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:01.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb0105700 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f2fa0000c00 tx=0x7f2fa000b040).stop 2023-12-26T22:00:01.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 shutdown_connections 2023-12-26T22:00:01.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f9c0310e0 0x7f2f9c0335a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:01.914 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 --2- 172.21.15.76:0/412239882 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2fb00f5a60 0x7f2fb0105700 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:01.914 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 >> 172.21.15.76:0/412239882 conn(0x7f2fb00f0390 msgr2=0x7f2fb00f0e80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:01.914 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 shutdown_connections 2023-12-26T22:00:01.914 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:01.913+0000 7f2fb877f700 1 -- 172.21.15.76:0/412239882 wait complete. 2023-12-26T22:00:01.914 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:03.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:01.353766+0000 mgr.smithi059.gwkiyc (mgr.14156) 100 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:03.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:01.912625+0000 mon.smithi059 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.76:0/412239882' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:03.312 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:03.312 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:04.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 -- 172.21.15.76:0/3825169694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 msgr2=0x7f42100f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:04.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 --2- 172.21.15.76:0/3825169694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f42100f6800 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f4204009fb0 tx=0x7f4204009b30).stop 2023-12-26T22:00:04.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 -- 172.21.15.76:0/3825169694 shutdown_connections 2023-12-26T22:00:04.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 --2- 172.21.15.76:0/3825169694 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f42100f6800 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:04.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 -- 172.21.15.76:0/3825169694 >> 172.21.15.76:0/3825169694 conn(0x7f42100f1b40 msgr2=0x7f42100f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 -- 172.21.15.76:0/3825169694 shutdown_connections 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.544+0000 7f420f59e700 1 -- 172.21.15.76:0/3825169694 wait complete. 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420f59e700 1 Processor -- start 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420f59e700 1 -- start start 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420f59e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f4210106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420f59e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4210106b10 con 0x7f42100f63e0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f4210106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f4210106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:57936/0 (socket says 172.21.15.76:57936) 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420e59c700 1 -- 172.21.15.76:0/1661912084 learned_addr learned my addr 172.21.15.76:0/1661912084 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420e59c700 1 -- 172.21.15.76:0/1661912084 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4204005040 con 0x7f42100f63e0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.545+0000 7f420e59c700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f4210106620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f4204000c00 tx=0x7f420400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.546+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f42040095f0 con 0x7f42100f63e0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.546+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4204007e50 con 0x7f42100f63e0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.546+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4210106d10 con 0x7f42100f63e0 2023-12-26T22:00:04.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.546+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4204014e70 con 0x7f42100f63e0 2023-12-26T22:00:04.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.546+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f42101071a0 con 0x7f42100f63e0 2023-12-26T22:00:04.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.546+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4204025070 con 0x7f42100f63e0 2023-12-26T22:00:04.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.547+0000 7f41feffd700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f41f8031090 0x7f41f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:04.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.547+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4210045bb0 con 0x7f42100f63e0 2023-12-26T22:00:04.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.547+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4204041dd0 con 0x7f42100f63e0 2023-12-26T22:00:04.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.547+0000 7f420dd9b700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f41f8031090 0x7f41f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:04.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.547+0000 7f420dd9b700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f41f8031090 0x7f41f8033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f42000096a0 tx=0x7f4200006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:04.550 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.552+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f42040079c0 con 0x7f42100f63e0 2023-12-26T22:00:04.758 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.759+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4210107fa0 con 0x7f42100f63e0 2023-12-26T22:00:04.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.759+0000 7f41feffd700 1 -- 172.21.15.76:0/1661912084 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f420400f300 con 0x7f42100f63e0 2023-12-26T22:00:04.759 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:04.759 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:04.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f41f8031090 msgr2=0x7f41f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:04.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f41f8031090 0x7f41f8033550 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f42000096a0 tx=0x7f4200006b40).stop 2023-12-26T22:00:04.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 msgr2=0x7f4210106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f4210106620 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f4204000c00 tx=0x7f420400b040).stop 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 shutdown_connections 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f41f8031090 0x7f41f8033550 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 --2- 172.21.15.76:0/1661912084 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f42100f63e0 0x7f4210106620 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 >> 172.21.15.76:0/1661912084 conn(0x7f42100f1b40 msgr2=0x7f42100f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 shutdown_connections 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:04.761+0000 7f420f59e700 1 -- 172.21.15.76:0/1661912084 wait complete. 2023-12-26T22:00:04.761 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:05.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:03.354114+0000 mgr.smithi059.gwkiyc (mgr.14156) 101 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:05.067 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:04.760010+0000 mon.smithi059 (mon.0) 175 : audit [DBG] from='client.? 172.21.15.76:0/1661912084' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:06.164 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:06.165 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:07.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:05.354524+0000 mgr.smithi059.gwkiyc (mgr.14156) 102 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 -- 172.21.15.76:0/2027016719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 msgr2=0x7fc83c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 --2- 172.21.15.76:0/2027016719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c0f56f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fc830009fb0 tx=0x7fc830009b30).stop 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 -- 172.21.15.76:0/2027016719 shutdown_connections 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 --2- 172.21.15.76:0/2027016719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c0f56f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 -- 172.21.15.76:0/2027016719 >> 172.21.15.76:0/2027016719 conn(0x7fc83c0f0a30 msgr2=0x7fc83c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 -- 172.21.15.76:0/2027016719 shutdown_connections 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.357+0000 7fc840b7c700 1 -- 172.21.15.76:0/2027016719 wait complete. 2023-12-26T22:00:07.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc840b7c700 1 Processor -- start 2023-12-26T22:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc840b7c700 1 -- start start 2023-12-26T22:00:07.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc840b7c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc840b7c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc83c105a00 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc83b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc83b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:36700/0 (socket says 172.21.15.76:36700) 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.358+0000 7fc83b7fe700 1 -- 172.21.15.76:0/3170200446 learned_addr learned my addr 172.21.15.76:0/3170200446 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc83b7fe700 1 -- 172.21.15.76:0/3170200446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc830005040 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc83b7fe700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c105510 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc830000c00 tx=0x7fc83000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc8300095f0 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc830007e50 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc830014e70 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc83c105c00 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.359+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc83c106090 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.360+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc830025070 con 0x7fc83c0f52d0 2023-12-26T22:00:07.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.360+0000 7fc823fff700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc824031090 0x7fc824033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.360+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc83c045bb0 con 0x7fc83c0f52d0 2023-12-26T22:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.360+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc830041e00 con 0x7fc83c0f52d0 2023-12-26T22:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.360+0000 7fc83affd700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc824031090 0x7fc824033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:07.361 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.361+0000 7fc83affd700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc824031090 0x7fc824033550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc82c0096a0 tx=0x7fc82c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:07.364 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.365+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc8300079c0 con 0x7fc83c0f52d0 2023-12-26T22:00:07.571 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.572+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc83c106e90 con 0x7fc83c0f52d0 2023-12-26T22:00:07.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.573+0000 7fc823fff700 1 -- 172.21.15.76:0/3170200446 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc83000f300 con 0x7fc83c0f52d0 2023-12-26T22:00:07.572 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:07.572 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc824031090 msgr2=0x7fc824033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc824031090 0x7fc824033550 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7fc82c0096a0 tx=0x7fc82c006b40).stop 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 msgr2=0x7fc83c105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c105510 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fc830000c00 tx=0x7fc83000b040).stop 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 shutdown_connections 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc824031090 0x7fc824033550 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 --2- 172.21.15.76:0/3170200446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc83c0f52d0 0x7fc83c105510 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.574+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 >> 172.21.15.76:0/3170200446 conn(0x7fc83c0f0a30 msgr2=0x7fc83c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.575+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 shutdown_connections 2023-12-26T22:00:07.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:07.575+0000 7fc840b7c700 1 -- 172.21.15.76:0/3170200446 wait complete. 2023-12-26T22:00:07.575 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:08.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:07 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:07.573462+0000 mon.smithi059 (mon.0) 176 : audit [DBG] from='client.? 172.21.15.76:0/3170200446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:08.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:08.963 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:09.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:07.354906+0000 mgr.smithi059.gwkiyc (mgr.14156) 103 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 -- 172.21.15.76:0/3525865442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 msgr2=0x7fad140f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 --2- 172.21.15.76:0/3525865442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad140f69b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7fad10004760 tx=0x7fad10009ad0).stop 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 -- 172.21.15.76:0/3525865442 shutdown_connections 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 --2- 172.21.15.76:0/3525865442 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad140f69b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 -- 172.21.15.76:0/3525865442 >> 172.21.15.76:0/3525865442 conn(0x7fad140f1cd0 msgr2=0x7fad140f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 -- 172.21.15.76:0/3525865442 shutdown_connections 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.149+0000 7fad1c1ab700 1 -- 172.21.15.76:0/3525865442 wait complete. 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad1c1ab700 1 Processor -- start 2023-12-26T22:00:10.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad1c1ab700 1 -- start start 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad1c1ab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad141067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad1c1ab700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad14106cb0 con 0x7fad140f6590 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad19f47700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad141067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad19f47700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad141067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:36730/0 (socket says 172.21.15.76:36730) 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.150+0000 7fad19f47700 1 -- 172.21.15.76:0/15214282 learned_addr learned my addr 172.21.15.76:0/15214282 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad19f47700 1 -- 172.21.15.76:0/15214282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad10009d70 con 0x7fad140f6590 2023-12-26T22:00:10.151 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad19f47700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad141067c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fad10013f10 tx=0x7fad10009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad1000e040 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad14106eb0 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad10007df0 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad10014d60 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.151+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad14107340 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.152+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fad10014520 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.152+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad14045bb0 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.152+0000 7fad0a7fc700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad00031090 0x7fad00033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.152+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fad10042580 con 0x7fad140f6590 2023-12-26T22:00:10.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.152+0000 7fad19746700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad00031090 0x7fad00033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:10.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.153+0000 7fad19746700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad00031090 0x7fad00033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fad040096a0 tx=0x7fad04006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:10.156 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.157+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad10007960 con 0x7fad140f6590 2023-12-26T22:00:10.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.363+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fad141081f0 con 0x7fad140f6590 2023-12-26T22:00:10.363 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.364+0000 7fad0a7fc700 1 -- 172.21.15.76:0/15214282 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad100262f0 con 0x7fad140f6590 2023-12-26T22:00:10.363 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:10.363 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.365+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad00031090 msgr2=0x7fad00033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.365+0000 7fad1c1ab700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad00031090 0x7fad00033550 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fad040096a0 tx=0x7fad04006b40).stop 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.365+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 msgr2=0x7fad141067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.365+0000 7fad1c1ab700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad141067c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fad10013f10 tx=0x7fad10009ad0).stop 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.366+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 shutdown_connections 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.366+0000 7fad1c1ab700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad00031090 0x7fad00033550 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.366+0000 7fad1c1ab700 1 --2- 172.21.15.76:0/15214282 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fad140f6590 0x7fad141067c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.366+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 >> 172.21.15.76:0/15214282 conn(0x7fad140f1cd0 msgr2=0x7fad140f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:10.365 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.366+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 shutdown_connections 2023-12-26T22:00:10.366 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:10.366+0000 7fad1c1ab700 1 -- 172.21.15.76:0/15214282 wait complete. 2023-12-26T22:00:10.366 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:11.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:09.355261+0000 mgr.smithi059.gwkiyc (mgr.14156) 104 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:11.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:10.364595+0000 mon.smithi059 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.76:0/15214282' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:11.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:11.749 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:12.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 -- 172.21.15.76:0/893471308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 msgr2=0x7f43b80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:12.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 --2- 172.21.15.76:0/893471308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b80f58a0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f43a8009fb0 tx=0x7f43a8009b30).stop 2023-12-26T22:00:12.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 -- 172.21.15.76:0/893471308 shutdown_connections 2023-12-26T22:00:12.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 --2- 172.21.15.76:0/893471308 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b80f58a0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 -- 172.21.15.76:0/893471308 >> 172.21.15.76:0/893471308 conn(0x7f43b80f0be0 msgr2=0x7f43b80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 -- 172.21.15.76:0/893471308 shutdown_connections 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.951+0000 7f43bf6ec700 1 -- 172.21.15.76:0/893471308 wait complete. 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bf6ec700 1 Processor -- start 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bf6ec700 1 -- start start 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bf6ec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b81094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:12.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bf6ec700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43b8060320 con 0x7f43b80f5480 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bd488700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b81094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bd488700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b81094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:36748/0 (socket says 172.21.15.76:36748) 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.952+0000 7f43bd488700 1 -- 172.21.15.76:0/1713160338 learned_addr learned my addr 172.21.15.76:0/1713160338 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43bd488700 1 -- 172.21.15.76:0/1713160338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a8005040 con 0x7f43b80f5480 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43bd488700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b81094f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f43a80089f0 tx=0x7f43a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a8009320 con 0x7f43b80f5480 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43b8060520 con 0x7f43b80f5480 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f43a8013ec0 con 0x7f43b80f5480 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43a8014d30 con 0x7f43b80f5480 2023-12-26T22:00:12.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.953+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43b8109ba0 con 0x7f43b80f5480 2023-12-26T22:00:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.954+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f43a80263a0 con 0x7f43b80f5480 2023-12-26T22:00:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.954+0000 7f43adffb700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f43a40310e0 0x7f43a40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.954+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f43a8042470 con 0x7f43b80f5480 2023-12-26T22:00:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.954+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43b8045bb0 con 0x7f43b80f5480 2023-12-26T22:00:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.954+0000 7f43bcc87700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f43a40310e0 0x7f43a40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:12.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.955+0000 7f43bcc87700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f43a40310e0 0x7f43a40335a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f43b40096a0 tx=0x7f43b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:12.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:12.959+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f43a80266d0 con 0x7f43b80f5480 2023-12-26T22:00:13.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:11.355674+0000 mgr.smithi059.gwkiyc (mgr.14156) 105 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:13.165 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.166+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43b80fb060 con 0x7f43b80f5480 2023-12-26T22:00:13.166 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.167+0000 7f43adffb700 1 -- 172.21.15.76:0/1713160338 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f43a8012330 con 0x7f43b80f5480 2023-12-26T22:00:13.166 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:13.166 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f43a40310e0 msgr2=0x7f43a40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f43a40310e0 0x7f43a40335a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f43b40096a0 tx=0x7f43b4006b40).stop 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 msgr2=0x7f43b81094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b81094f0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f43a80089f0 tx=0x7f43a800b040).stop 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 shutdown_connections 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f43a40310e0 0x7f43a40335a0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 --2- 172.21.15.76:0/1713160338 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f43b80f5480 0x7f43b81094f0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 >> 172.21.15.76:0/1713160338 conn(0x7f43b80f0be0 msgr2=0x7f43b80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:13.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 shutdown_connections 2023-12-26T22:00:13.169 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:13.168+0000 7f43bf6ec700 1 -- 172.21.15.76:0/1713160338 wait complete. 2023-12-26T22:00:13.169 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:14.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:13 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:13.167293+0000 mon.smithi059 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.76:0/1713160338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:14.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:14.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:15.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:13.355990+0000 mgr.smithi059.gwkiyc (mgr.14156) 106 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:15.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 -- 172.21.15.76:0/2216009723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 msgr2=0x7f13380f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:15.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 --2- 172.21.15.76:0/2216009723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f13380f56f0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f1328009fb0 tx=0x7f1328009b30).stop 2023-12-26T22:00:15.739 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 -- 172.21.15.76:0/2216009723 shutdown_connections 2023-12-26T22:00:15.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 --2- 172.21.15.76:0/2216009723 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f13380f56f0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:15.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 -- 172.21.15.76:0/2216009723 >> 172.21.15.76:0/2216009723 conn(0x7f13380f0a10 msgr2=0x7f13380f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:15.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 -- 172.21.15.76:0/2216009723 shutdown_connections 2023-12-26T22:00:15.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 -- 172.21.15.76:0/2216009723 wait complete. 2023-12-26T22:00:15.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.740+0000 7f133d034700 1 Processor -- start 2023-12-26T22:00:15.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f133d034700 1 -- start start 2023-12-26T22:00:15.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f133d034700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f1338105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:15.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f133d034700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1338105a30 con 0x7f13380f52d0 2023-12-26T22:00:15.741 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f1337fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f1338105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f1337fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f1338105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39254/0 (socket says 172.21.15.76:39254) 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f1337fff700 1 -- 172.21.15.76:0/431879155 learned_addr learned my addr 172.21.15.76:0/431879155 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f1337fff700 1 -- 172.21.15.76:0/431879155 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1328005040 con 0x7f13380f52d0 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.741+0000 7f1337fff700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f1338105540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f13280089f0 tx=0x7f132800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.742+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1328009320 con 0x7f13380f52d0 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.742+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1338105c30 con 0x7f13380f52d0 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.742+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1328013ec0 con 0x7f13380f52d0 2023-12-26T22:00:15.745 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.742+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1328014d30 con 0x7f13380f52d0 2023-12-26T22:00:15.746 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.742+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13381060c0 con 0x7f13380f52d0 2023-12-26T22:00:15.746 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.743+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f13280095f0 con 0x7f13380f52d0 2023-12-26T22:00:15.746 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.743+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1338045bb0 con 0x7f13380f52d0 2023-12-26T22:00:15.747 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.743+0000 7f1334ff9700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1320031090 0x7f1320033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:15.747 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.743+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13280423e0 con 0x7f13380f52d0 2023-12-26T22:00:15.747 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.743+0000 7f13377fe700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1320031090 0x7f1320033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:15.747 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.744+0000 7f13377fe700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1320031090 0x7f1320033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f132c0096a0 tx=0x7f132c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:15.747 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.748+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f13280268c0 con 0x7f13380f52d0 2023-12-26T22:00:15.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.956+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1338106f70 con 0x7f13380f52d0 2023-12-26T22:00:15.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.957+0000 7f1334ff9700 1 -- 172.21.15.76:0/431879155 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f132801d7f0 con 0x7f13380f52d0 2023-12-26T22:00:15.956 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:15.956 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1320031090 msgr2=0x7f1320033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1320031090 0x7f1320033550 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f132c0096a0 tx=0x7f132c006b40).stop 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 msgr2=0x7f1338105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f1338105540 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f13280089f0 tx=0x7f132800b040).stop 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 shutdown_connections 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1320031090 0x7f1320033550 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 --2- 172.21.15.76:0/431879155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13380f52d0 0x7f1338105540 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 >> 172.21.15.76:0/431879155 conn(0x7f13380f0a10 msgr2=0x7f13380f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 shutdown_connections 2023-12-26T22:00:15.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:15.958+0000 7f133d034700 1 -- 172.21.15.76:0/431879155 wait complete. 2023-12-26T22:00:15.959 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:17.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:15.356342+0000 mgr.smithi059.gwkiyc (mgr.14156) 107 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:17.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:15.957320+0000 mon.smithi059 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.76:0/431879155' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:17.346 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:17.346 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 -- 172.21.15.76:0/2763160850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 msgr2=0x7fa2fc0f91e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 --2- 172.21.15.76:0/2763160850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc0f91e0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fa2ec009fb0 tx=0x7fa2ec009b30).stop 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 -- 172.21.15.76:0/2763160850 shutdown_connections 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 --2- 172.21.15.76:0/2763160850 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc0f91e0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 -- 172.21.15.76:0/2763160850 >> 172.21.15.76:0/2763160850 conn(0x7fa2fc0f0be0 msgr2=0x7fa2fc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 -- 172.21.15.76:0/2763160850 shutdown_connections 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.544+0000 7fa303825700 1 -- 172.21.15.76:0/2763160850 wait complete. 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa303825700 1 Processor -- start 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa303825700 1 -- start start 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa303825700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:18.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa303825700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2fc106c70 con 0x7fa2fc0602e0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa3015c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa3015c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39264/0 (socket says 172.21.15.76:39264) 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa3015c1700 1 -- 172.21.15.76:0/2258539964 learned_addr learned my addr 172.21.15.76:0/2258539964 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa3015c1700 1 -- 172.21.15.76:0/2258539964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2ec005040 con 0x7fa2fc0602e0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa3015c1700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc106780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fa2ec009f80 tx=0x7fa2ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.545+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec009320 con 0x7fa2fc0602e0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.546+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2fc106e70 con 0x7fa2fc0602e0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.546+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa2ec013680 con 0x7fa2fc0602e0 2023-12-26T22:00:18.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.546+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2ec01eda0 con 0x7fa2fc0602e0 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.546+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2fc107300 con 0x7fa2fc0602e0 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.546+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa2ec01e420 con 0x7fa2fc0602e0 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.547+0000 7fa2f1ffb700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2e8031090 0x7fa2e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.547+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2ec0423f0 con 0x7fa2fc0602e0 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.547+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2fc045bb0 con 0x7fa2fc0602e0 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.547+0000 7fa300dc0700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2e8031090 0x7fa2e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:18.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.547+0000 7fa300dc0700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2e8031090 0x7fa2e8033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa2f80096a0 tx=0x7fa2f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:18.552 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.552+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2ec01e700 con 0x7fa2fc0602e0 2023-12-26T22:00:18.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.759+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2fc0fc150 con 0x7fa2fc0602e0 2023-12-26T22:00:18.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.760+0000 7fa2f1ffb700 1 -- 172.21.15.76:0/2258539964 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2ec01cb10 con 0x7fa2fc0602e0 2023-12-26T22:00:18.761 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:18.761 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2e8031090 msgr2=0x7fa2e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2e8031090 0x7fa2e8033550 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fa2f80096a0 tx=0x7fa2f8006b40).stop 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 msgr2=0x7fa2fc106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc106780 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7fa2ec009f80 tx=0x7fa2ec00b040).stop 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 shutdown_connections 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2e8031090 0x7fa2e8033550 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:18.762 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 --2- 172.21.15.76:0/2258539964 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2fc0602e0 0x7fa2fc106780 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:18.763 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 >> 172.21.15.76:0/2258539964 conn(0x7fa2fc0f0be0 msgr2=0x7fa2fc0f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:18.763 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 shutdown_connections 2023-12-26T22:00:18.763 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:18.762+0000 7fa303825700 1 -- 172.21.15.76:0/2258539964 wait complete. 2023-12-26T22:00:18.763 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:19.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:17.356716+0000 mgr.smithi059.gwkiyc (mgr.14156) 108 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:19.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:18.761710+0000 mon.smithi059 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.76:0/2258539964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:20.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:20.147 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:21.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:19.357116+0000 mgr.smithi059.gwkiyc (mgr.14156) 109 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:21.282 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.280+0000 7f488133a700 1 -- 172.21.15.76:0/739623921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 msgr2=0x7f487c0f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.280+0000 7f488133a700 1 --2- 172.21.15.76:0/739623921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c0f5030 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f486c004770 tx=0x7f486c00ac40).stop 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 -- 172.21.15.76:0/739623921 shutdown_connections 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 --2- 172.21.15.76:0/739623921 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c0f5030 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 -- 172.21.15.76:0/739623921 >> 172.21.15.76:0/739623921 conn(0x7f487c0f0390 msgr2=0x7f487c0f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 -- 172.21.15.76:0/739623921 shutdown_connections 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 -- 172.21.15.76:0/739623921 wait complete. 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 Processor -- start 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 -- start start 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.281+0000 7f488133a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f487c106cc0 con 0x7f487c0f4c10 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f487affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:21.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f487affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39278/0 (socket says 172.21.15.76:39278) 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f487affd700 1 -- 172.21.15.76:0/3867251181 learned_addr learned my addr 172.21.15.76:0/3867251181 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f487affd700 1 -- 172.21.15.76:0/3867251181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f486c005040 con 0x7f487c0f4c10 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f487affd700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c1067d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f486c004740 tx=0x7f486c00ac40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486c019070 con 0x7f487c0f4c10 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f487c106ec0 con 0x7f487c0f4c10 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f486c0077b0 con 0x7f487c0f4c10 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f486c008200 con 0x7f487c0f4c10 2023-12-26T22:00:21.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.282+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f487c107350 con 0x7f487c0f4c10 2023-12-26T22:00:21.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.283+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f486c0084c0 con 0x7f487c0f4c10 2023-12-26T22:00:21.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.283+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f487c045bb0 con 0x7f487c0f4c10 2023-12-26T22:00:21.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.283+0000 7f48637fe700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4864031040 0x7f4864033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:21.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.283+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f486c042000 con 0x7f487c0f4c10 2023-12-26T22:00:21.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.283+0000 7f487a7fc700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4864031040 0x7f4864033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:21.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.284+0000 7f487a7fc700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4864031040 0x7f4864033500 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f48700096a0 tx=0x7f4870006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:21.288 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.288+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f486c01c730 con 0x7f487c0f4c10 2023-12-26T22:00:21.498 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.498+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f487c1081a0 con 0x7f487c0f4c10 2023-12-26T22:00:21.498 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.498+0000 7f48637fe700 1 -- 172.21.15.76:0/3867251181 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f486c01c320 con 0x7f487c0f4c10 2023-12-26T22:00:21.499 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:21.499 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:21.500 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4864031040 msgr2=0x7f4864033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:21.500 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4864031040 0x7f4864033500 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f48700096a0 tx=0x7f4870006b40).stop 2023-12-26T22:00:21.500 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 msgr2=0x7f487c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c1067d0 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f486c004740 tx=0x7f486c00ac40).stop 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 shutdown_connections 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4864031040 0x7f4864033500 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 --2- 172.21.15.76:0/3867251181 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f487c0f4c10 0x7f487c1067d0 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 >> 172.21.15.76:0/3867251181 conn(0x7f487c0f0390 msgr2=0x7f487c0f0e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 shutdown_connections 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:21.500+0000 7f488133a700 1 -- 172.21.15.76:0/3867251181 wait complete. 2023-12-26T22:00:21.501 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:22.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:21 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:21.499843+0000 mon.smithi059 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.76:0/3867251181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:22.890 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:22.890 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:23.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:21.357536+0000 mgr.smithi059.gwkiyc (mgr.14156) 110 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 -- 172.21.15.76:0/645225603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 msgr2=0x7efe5c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:24.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 --2- 172.21.15.76:0/645225603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c0f56f0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7efe4c009fb0 tx=0x7efe4c009b30).stop 2023-12-26T22:00:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 -- 172.21.15.76:0/645225603 shutdown_connections 2023-12-26T22:00:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 --2- 172.21.15.76:0/645225603 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c0f56f0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 -- 172.21.15.76:0/645225603 >> 172.21.15.76:0/645225603 conn(0x7efe5c0f0a30 msgr2=0x7efe5c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 -- 172.21.15.76:0/645225603 shutdown_connections 2023-12-26T22:00:24.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.095+0000 7efe5bfff700 1 -- 172.21.15.76:0/645225603 wait complete. 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5bfff700 1 Processor -- start 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5bfff700 1 -- start start 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c1092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5bfff700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe5c060360 con 0x7efe5c0f52d0 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c1092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5affd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c1092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39286/0 (socket says 172.21.15.76:39286) 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5affd700 1 -- 172.21.15.76:0/664044393 learned_addr learned my addr 172.21.15.76:0/664044393 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:24.098 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.096+0000 7efe5affd700 1 -- 172.21.15.76:0/664044393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe4c005040 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.097+0000 7efe5affd700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c1092d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7efe4c0089f0 tx=0x7efe4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.097+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4c009320 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.097+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe5c060560 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.097+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efe4c013ec0 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.097+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe4c014d30 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.097+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe5c109a00 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.098+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efe4c014420 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.098+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe5c045bb0 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.098+0000 7efe437fe700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe44031090 0x7efe44033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.098+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efe4c0424a0 con 0x7efe5c0f52d0 2023-12-26T22:00:24.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.098+0000 7efe5a7fc700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe44031090 0x7efe44033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:24.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.099+0000 7efe5a7fc700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe44031090 0x7efe44033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efe500096a0 tx=0x7efe50006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:24.103 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.103+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efe4c01d7f0 con 0x7efe5c0f52d0 2023-12-26T22:00:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.307+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe5c0fadc0 con 0x7efe5c0f52d0 2023-12-26T22:00:24.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.308+0000 7efe437fe700 1 -- 172.21.15.76:0/664044393 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe4c01d7f0 con 0x7efe5c0f52d0 2023-12-26T22:00:24.308 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:24.308 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:24.309 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe44031090 msgr2=0x7efe44033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:24.309 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe44031090 0x7efe44033550 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7efe500096a0 tx=0x7efe50006b40).stop 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 msgr2=0x7efe5c1092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c1092d0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7efe4c0089f0 tx=0x7efe4c00b040).stop 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 shutdown_connections 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe44031090 0x7efe44033550 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 --2- 172.21.15.76:0/664044393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe5c0f52d0 0x7efe5c1092d0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 >> 172.21.15.76:0/664044393 conn(0x7efe5c0f0a30 msgr2=0x7efe5c0f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 shutdown_connections 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:24.309+0000 7efe5bfff700 1 -- 172.21.15.76:0/664044393 wait complete. 2023-12-26T22:00:24.310 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:25.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:23.357842+0000 mgr.smithi059.gwkiyc (mgr.14156) 111 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:25.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:24.309120+0000 mon.smithi059 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.76:0/664044393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:25.710 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:25.711 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:26.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.881+0000 7fec5e23d700 1 -- 172.21.15.76:0/4103133342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 msgr2=0x7fec580f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:26.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.881+0000 7fec5e23d700 1 --2- 172.21.15.76:0/4103133342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec580f6800 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fec54009fb0 tx=0x7fec54009b30).stop 2023-12-26T22:00:26.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.881+0000 7fec5e23d700 1 -- 172.21.15.76:0/4103133342 shutdown_connections 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.881+0000 7fec5e23d700 1 --2- 172.21.15.76:0/4103133342 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec580f6800 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.881+0000 7fec5e23d700 1 -- 172.21.15.76:0/4103133342 >> 172.21.15.76:0/4103133342 conn(0x7fec580f1b40 msgr2=0x7fec580f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.882+0000 7fec5e23d700 1 -- 172.21.15.76:0/4103133342 shutdown_connections 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.882+0000 7fec5e23d700 1 -- 172.21.15.76:0/4103133342 wait complete. 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.882+0000 7fec5e23d700 1 Processor -- start 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.882+0000 7fec5e23d700 1 -- start start 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5e23d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec58106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5e23d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec58106b10 con 0x7fec580f63e0 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5d23b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec58106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5d23b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec58106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55350/0 (socket says 172.21.15.76:55350) 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5d23b700 1 -- 172.21.15.76:0/3439960018 learned_addr learned my addr 172.21.15.76:0/3439960018 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5d23b700 1 -- 172.21.15.76:0/3439960018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec54005040 con 0x7fec580f63e0 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5d23b700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec58106620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fec54000c00 tx=0x7fec5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec540095f0 con 0x7fec580f63e0 2023-12-26T22:00:26.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fec54007e50 con 0x7fec580f63e0 2023-12-26T22:00:26.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec58106d10 con 0x7fec580f63e0 2023-12-26T22:00:26.885 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.883+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec54014e70 con 0x7fec580f63e0 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.884+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec581071a0 con 0x7fec580f63e0 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.884+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fec54025070 con 0x7fec580f63e0 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.885+0000 7fec4dffb700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec44031090 0x7fec44033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.885+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fec54041e20 con 0x7fec580f63e0 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.885+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec58045bb0 con 0x7fec580f63e0 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.885+0000 7fec5ca3a700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec44031090 0x7fec44033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:26.886 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.885+0000 7fec5ca3a700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec44031090 0x7fec44033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fec480096a0 tx=0x7fec48006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:26.890 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:26.890+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fec5400f4e0 con 0x7fec580f63e0 2023-12-26T22:00:27.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:25.358155+0000 mgr.smithi059.gwkiyc (mgr.14156) 112 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:27.097 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.096+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fec580fbfc0 con 0x7fec580f63e0 2023-12-26T22:00:27.097 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.097+0000 7fec4dffb700 1 -- 172.21.15.76:0/3439960018 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec5402d090 con 0x7fec580f63e0 2023-12-26T22:00:27.098 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:27.098 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:27.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec44031090 msgr2=0x7fec44033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:27.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec44031090 0x7fec44033550 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fec480096a0 tx=0x7fec48006b40).stop 2023-12-26T22:00:27.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 msgr2=0x7fec58106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:27.099 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec58106620 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fec54000c00 tx=0x7fec5400b040).stop 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 shutdown_connections 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec44031090 0x7fec44033550 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 --2- 172.21.15.76:0/3439960018 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fec580f63e0 0x7fec58106620 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 >> 172.21.15.76:0/3439960018 conn(0x7fec580f1b40 msgr2=0x7fec580f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 shutdown_connections 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:27.099+0000 7fec5e23d700 1 -- 172.21.15.76:0/3439960018 wait complete. 2023-12-26T22:00:27.100 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:28.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:27 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:27.098874+0000 mon.smithi059 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.76:0/3439960018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:28.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:28.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:29.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:27.358445+0000 mgr.smithi059.gwkiyc (mgr.14156) 113 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:29.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 -- 172.21.15.76:0/2004221761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 msgr2=0x7f65900f5e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:29.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 --2- 172.21.15.76:0/2004221761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65900f5e70 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f658c009fb0 tx=0x7f658c009b30).stop 2023-12-26T22:00:29.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 -- 172.21.15.76:0/2004221761 shutdown_connections 2023-12-26T22:00:29.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 --2- 172.21.15.76:0/2004221761 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65900f5e70 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:29.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 -- 172.21.15.76:0/2004221761 >> 172.21.15.76:0/2004221761 conn(0x7f65900f0360 msgr2=0x7f65900f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:29.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 -- 172.21.15.76:0/2004221761 shutdown_connections 2023-12-26T22:00:29.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.680+0000 7f65984c4700 1 -- 172.21.15.76:0/2004221761 wait complete. 2023-12-26T22:00:29.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f65984c4700 1 Processor -- start 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f65984c4700 1 -- start start 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f65984c4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65901094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f65984c4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65900f5470 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f6596260700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65901094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f6596260700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65901094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55356/0 (socket says 172.21.15.76:55356) 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f6596260700 1 -- 172.21.15.76:0/4207975393 learned_addr learned my addr 172.21.15.76:0/4207975393 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.681+0000 7f6596260700 1 -- 172.21.15.76:0/4207975393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c005040 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.682+0000 7f6596260700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65901094b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f658c000c00 tx=0x7f658c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.682+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658c0095f0 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.682+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65900f5670 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.682+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f658c007e50 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.682+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f658c014e70 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.682+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6590109be0 con 0x7f65900f5a50 2023-12-26T22:00:29.684 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.683+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f658c01d4e0 con 0x7f65900f5a50 2023-12-26T22:00:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.683+0000 7f6586ffd700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f657c031090 0x7f657c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.683+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f658c041d50 con 0x7f65900f5a50 2023-12-26T22:00:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.683+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6590045bb0 con 0x7f65900f5a50 2023-12-26T22:00:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.683+0000 7f6595a5f700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f657c031090 0x7f657c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:29.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.683+0000 7f6595a5f700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f657c031090 0x7f657c033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f65800096a0 tx=0x7f6580006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:29.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.688+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f658c01b8b0 con 0x7f65900f5a50 2023-12-26T22:00:29.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.896+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f65900fb060 con 0x7f65900f5a50 2023-12-26T22:00:29.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.897+0000 7f6586ffd700 1 -- 172.21.15.76:0/4207975393 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f658c01b2c0 con 0x7f65900f5a50 2023-12-26T22:00:29.897 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:29.898 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.898+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f657c031090 msgr2=0x7f657c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.898+0000 7f65984c4700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f657c031090 0x7f657c033550 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f65800096a0 tx=0x7f6580006b40).stop 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.898+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 msgr2=0x7f65901094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.898+0000 7f65984c4700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65901094b0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f658c000c00 tx=0x7f658c00b040).stop 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.899+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 shutdown_connections 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.899+0000 7f65984c4700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f657c031090 0x7f657c033550 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.899+0000 7f65984c4700 1 --2- 172.21.15.76:0/4207975393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f65900f5a50 0x7f65901094b0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.899+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 >> 172.21.15.76:0/4207975393 conn(0x7f65900f0360 msgr2=0x7f65900f0e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.899+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 shutdown_connections 2023-12-26T22:00:29.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:29.899+0000 7f65984c4700 1 -- 172.21.15.76:0/4207975393 wait complete. 2023-12-26T22:00:29.900 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:31.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:29.358718+0000 mgr.smithi059.gwkiyc (mgr.14156) 114 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:31.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:29.898598+0000 mon.smithi059 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.76:0/4207975393' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:31.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:31.295 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 -- 172.21.15.76:0/1597393302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 msgr2=0x7faf040f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 --2- 172.21.15.76:0/1597393302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040f58d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7faef4009fb0 tx=0x7faef4009b30).stop 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 -- 172.21.15.76:0/1597393302 shutdown_connections 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 --2- 172.21.15.76:0/1597393302 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040f58d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 -- 172.21.15.76:0/1597393302 >> 172.21.15.76:0/1597393302 conn(0x7faf040f0c10 msgr2=0x7faf040f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 -- 172.21.15.76:0/1597393302 shutdown_connections 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 -- 172.21.15.76:0/1597393302 wait complete. 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.503+0000 7faf0beed700 1 Processor -- start 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf0beed700 1 -- start start 2023-12-26T22:00:32.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf0beed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040fd490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf0beed700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf040fb910 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf09c89700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040fd490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf09c89700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040fd490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55368/0 (socket says 172.21.15.76:55368) 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf09c89700 1 -- 172.21.15.76:0/2782748359 learned_addr learned my addr 172.21.15.76:0/2782748359 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf09c89700 1 -- 172.21.15.76:0/2782748359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faef4005040 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf09c89700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040fd490 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7faef40089f0 tx=0x7faef400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef4009320 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf040fbab0 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7faef4013ec0 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faef4014d30 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.504+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf040fbf40 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.505+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7faef40095f0 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.505+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf040591e0 con 0x7faf040f54b0 2023-12-26T22:00:32.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.505+0000 7faefa7fc700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faef0031090 0x7faef0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.505+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7faef40424a0 con 0x7faf040f54b0 2023-12-26T22:00:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.506+0000 7faf09488700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faef0031090 0x7faef0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:32.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.506+0000 7faf09488700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faef0031090 0x7faef0033550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faf000096a0 tx=0x7faf00006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:32.513 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.510+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7faef401d7f0 con 0x7faf040f54b0 2023-12-26T22:00:32.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.717+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7faf040fafa0 con 0x7faf040f54b0 2023-12-26T22:00:32.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.718+0000 7faefa7fc700 1 -- 172.21.15.76:0/2782748359 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7faef401d7f0 con 0x7faf040f54b0 2023-12-26T22:00:32.718 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:32.718 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:32.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faef0031090 msgr2=0x7faef0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faef0031090 0x7faef0033550 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7faf000096a0 tx=0x7faf00006b40).stop 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 msgr2=0x7faf040fd490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040fd490 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7faef40089f0 tx=0x7faef400b040).stop 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 shutdown_connections 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7faef0031090 0x7faef0033550 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 --2- 172.21.15.76:0/2782748359 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7faf040f54b0 0x7faf040fd490 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 >> 172.21.15.76:0/2782748359 conn(0x7faf040f0c10 msgr2=0x7faf0405ff80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 shutdown_connections 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:32.719+0000 7faf0beed700 1 -- 172.21.15.76:0/2782748359 wait complete. 2023-12-26T22:00:32.720 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:33.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:31.359052+0000 mgr.smithi059.gwkiyc (mgr.14156) 115 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:33.045 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:32.719437+0000 mon.smithi059 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.76:0/2782748359' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:34.101 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:34.101 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:35.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:33.359453+0000 mgr.smithi059.gwkiyc (mgr.14156) 116 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- 172.21.15.76:0/3980523799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 msgr2=0x7f29400f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 --2- 172.21.15.76:0/3980523799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f29400f69b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f2934009fb0 tx=0x7f2934009b30).stop 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- 172.21.15.76:0/3980523799 shutdown_connections 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 --2- 172.21.15.76:0/3980523799 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f29400f69b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- 172.21.15.76:0/3980523799 >> 172.21.15.76:0/3980523799 conn(0x7f29400f1cf0 msgr2=0x7f29400f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- 172.21.15.76:0/3980523799 shutdown_connections 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- 172.21.15.76:0/3980523799 wait complete. 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 Processor -- start 2023-12-26T22:00:35.333 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- start start 2023-12-26T22:00:35.334 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f294006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:35.334 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.333+0000 7f2944ebe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2940069f00 con 0x7f29400f6590 2023-12-26T22:00:35.334 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f293e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f294006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:35.334 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f293e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f294006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55376/0 (socket says 172.21.15.76:55376) 2023-12-26T22:00:35.334 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f293e59c700 1 -- 172.21.15.76:0/2870614712 learned_addr learned my addr 172.21.15.76:0/2870614712 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:35.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f293e59c700 1 -- 172.21.15.76:0/2870614712 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2934005040 con 0x7f29400f6590 2023-12-26T22:00:35.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f293e59c700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f294006ba80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2934000c00 tx=0x7f293400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:35.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29340095f0 con 0x7f29400f6590 2023-12-26T22:00:35.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f294006a0a0 con 0x7f29400f6590 2023-12-26T22:00:35.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2934007e50 con 0x7f29400f6590 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2934014e70 con 0x7f29400f6590 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.334+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f294006a530 con 0x7f29400f6590 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.335+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2934025070 con 0x7f29400f6590 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.336+0000 7f292effd700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2928031090 0x7f2928033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.336+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29400591e0 con 0x7f29400f6590 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.336+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2934041d10 con 0x7f29400f6590 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.336+0000 7f293dd9b700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2928031090 0x7f2928033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:35.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.336+0000 7f293dd9b700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2928031090 0x7f2928033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f29300096a0 tx=0x7f2930006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:35.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.340+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f29340079c0 con 0x7f29400f6590 2023-12-26T22:00:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.547+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2940045bb0 con 0x7f29400f6590 2023-12-26T22:00:35.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.548+0000 7f292effd700 1 -- 172.21.15.76:0/2870614712 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f293402d090 con 0x7f29400f6590 2023-12-26T22:00:35.548 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:35.548 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2928031090 msgr2=0x7f2928033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2928031090 0x7f2928033550 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f29300096a0 tx=0x7f2930006b40).stop 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 msgr2=0x7f294006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f294006ba80 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f2934000c00 tx=0x7f293400b040).stop 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 shutdown_connections 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2928031090 0x7f2928033550 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 --2- 172.21.15.76:0/2870614712 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f29400f6590 0x7f294006ba80 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:35.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 >> 172.21.15.76:0/2870614712 conn(0x7f29400f1cf0 msgr2=0x7f29400f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 shutdown_connections 2023-12-26T22:00:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:35.550+0000 7f2944ebe700 1 -- 172.21.15.76:0/2870614712 wait complete. 2023-12-26T22:00:35.550 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:36.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:35 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:35.548759+0000 mon.smithi059 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.76:0/2870614712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:36.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:36.955 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:35.359717+0000 mgr.smithi059.gwkiyc (mgr.14156) 117 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/4255085846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 msgr2=0x7f6cdc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 --2- 172.21.15.76:0/4255085846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc0f69b0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f6cd0004760 tx=0x7f6cd0009ad0).stop 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/4255085846 shutdown_connections 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 --2- 172.21.15.76:0/4255085846 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc0f69b0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/4255085846 >> 172.21.15.76:0/4255085846 conn(0x7f6cdc0f1cd0 msgr2=0x7f6cdc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/4255085846 shutdown_connections 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/4255085846 wait complete. 2023-12-26T22:00:38.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 Processor -- start 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 -- start start 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.154+0000 7f6ce2ef6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce2ef6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cdc069f00 con 0x7f6cdc0f6590 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce0c92700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce0c92700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44206/0 (socket says 172.21.15.76:44206) 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce0c92700 1 -- 172.21.15.76:0/3413453102 learned_addr learned my addr 172.21.15.76:0/3413453102 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:38.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce0c92700 1 -- 172.21.15.76:0/3413453102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd0009d70 con 0x7f6cdc0f6590 2023-12-26T22:00:38.156 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce0c92700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc06ba80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f6cd0004170 tx=0x7f6cd0009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:38.156 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd000e040 con 0x7f6cdc0f6590 2023-12-26T22:00:38.156 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cdc06a0a0 con 0x7f6cdc0f6590 2023-12-26T22:00:38.156 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6cd0008ef0 con 0x7f6cdc0f6590 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cd0014c90 con 0x7f6cdc0f6590 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.155+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cdc06a530 con 0x7f6cdc0f6590 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.156+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6cd0014420 con 0x7f6cdc0f6590 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.156+0000 7f6cd97fa700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6ccc031090 0x7f6ccc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.156+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6cd0042220 con 0x7f6cdc0f6590 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.156+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cdc0591e0 con 0x7f6cdc0f6590 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.157+0000 7f6cdbfff700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6ccc031090 0x7f6ccc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:38.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.157+0000 7f6cdbfff700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6ccc031090 0x7f6ccc033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:38.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.161+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6cd0012d00 con 0x7f6cdc0f6590 2023-12-26T22:00:38.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.370+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cdc0fc080 con 0x7f6cdc0f6590 2023-12-26T22:00:38.370 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.371+0000 7f6cd97fa700 1 -- 172.21.15.76:0/3413453102 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cd0012b20 con 0x7f6cdc0f6590 2023-12-26T22:00:38.370 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:38.370 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6ccc031090 msgr2=0x7f6ccc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6ccc031090 0x7f6ccc033550 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).stop 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 msgr2=0x7f6cdc06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc06ba80 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f6cd0004170 tx=0x7f6cd0009ad0).stop 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 shutdown_connections 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6ccc031090 0x7f6ccc033550 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 --2- 172.21.15.76:0/3413453102 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6cdc0f6590 0x7f6cdc06ba80 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 >> 172.21.15.76:0/3413453102 conn(0x7f6cdc0f1cd0 msgr2=0x7f6cdc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 shutdown_connections 2023-12-26T22:00:38.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:38.372+0000 7f6ce2ef6700 1 -- 172.21.15.76:0/3413453102 wait complete. 2023-12-26T22:00:38.373 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:39.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:37.360051+0000 mgr.smithi059.gwkiyc (mgr.14156) 118 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:39.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:38.371435+0000 mon.smithi059 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.76:0/3413453102' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:39.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:39.752 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 -- 172.21.15.76:0/1299740050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 msgr2=0x7fed8c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 --2- 172.21.15.76:0/1299740050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c0f58a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fed7c009fb0 tx=0x7fed7c009b30).stop 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 -- 172.21.15.76:0/1299740050 shutdown_connections 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 --2- 172.21.15.76:0/1299740050 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c0f58a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 -- 172.21.15.76:0/1299740050 >> 172.21.15.76:0/1299740050 conn(0x7fed8c0f0be0 msgr2=0x7fed8c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 -- 172.21.15.76:0/1299740050 shutdown_connections 2023-12-26T22:00:40.964 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.964+0000 7fed9235c700 1 -- 172.21.15.76:0/1299740050 wait complete. 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.965+0000 7fed9235c700 1 Processor -- start 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.965+0000 7fed9235c700 1 -- start start 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.965+0000 7fed9235c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed9235c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c105be0 con 0x7fed8c0f5480 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed8bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44236/0 (socket says 172.21.15.76:44236) 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed8bfff700 1 -- 172.21.15.76:0/2063305171 learned_addr learned my addr 172.21.15.76:0/2063305171 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed8bfff700 1 -- 172.21.15.76:0/2063305171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7c005040 con 0x7fed8c0f5480 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed8bfff700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c1056f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fed7c000c00 tx=0x7fed7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed7c0095f0 con 0x7fed8c0f5480 2023-12-26T22:00:40.966 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fed7c007e50 con 0x7fed8c0f5480 2023-12-26T22:00:40.967 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed7c014e70 con 0x7fed8c0f5480 2023-12-26T22:00:40.967 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c105de0 con 0x7fed8c0f5480 2023-12-26T22:00:40.967 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.966+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c106270 con 0x7fed8c0f5480 2023-12-26T22:00:40.967 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.967+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fed7c025070 con 0x7fed8c0f5480 2023-12-26T22:00:40.968 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.967+0000 7fed88ff9700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fed74031090 0x7fed74033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:40.968 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.967+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c045bb0 con 0x7fed8c0f5480 2023-12-26T22:00:40.968 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.967+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fed7c041d20 con 0x7fed8c0f5480 2023-12-26T22:00:40.968 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.968+0000 7fed8b7fe700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fed74031090 0x7fed74033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:40.968 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.968+0000 7fed8b7fe700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fed74031090 0x7fed74033550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fed800096a0 tx=0x7fed80006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:40.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:40.972+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fed7c0079c0 con 0x7fed8c0f5480 2023-12-26T22:00:41.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.177+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed8c0fb060 con 0x7fed8c0f5480 2023-12-26T22:00:41.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.178+0000 7fed88ff9700 1 -- 172.21.15.76:0/2063305171 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fed7c00f300 con 0x7fed8c0f5480 2023-12-26T22:00:41.177 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:41.177 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.179+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fed74031090 msgr2=0x7fed74033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.179+0000 7fed9235c700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fed74031090 0x7fed74033550 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fed800096a0 tx=0x7fed80006b40).stop 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.179+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 msgr2=0x7fed8c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.179+0000 7fed9235c700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c1056f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7fed7c000c00 tx=0x7fed7c00b040).stop 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.180+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 shutdown_connections 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.180+0000 7fed9235c700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fed74031090 0x7fed74033550 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.180+0000 7fed9235c700 1 --2- 172.21.15.76:0/2063305171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fed8c0f5480 0x7fed8c1056f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.180+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 >> 172.21.15.76:0/2063305171 conn(0x7fed8c0f0be0 msgr2=0x7fed8c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.180+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 shutdown_connections 2023-12-26T22:00:41.179 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:41.180+0000 7fed9235c700 1 -- 172.21.15.76:0/2063305171 wait complete. 2023-12-26T22:00:41.180 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:39.360500+0000 mgr.smithi059.gwkiyc (mgr.14156) 119 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:42.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:41 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:41.178609+0000 mon.smithi059 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.76:0/2063305171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:42.577 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:42.578 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:41.360866+0000 mgr.smithi059.gwkiyc (mgr.14156) 120 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 -- 172.21.15.76:0/3921768174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d280f63e0 msgr2=0x7f5d280f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 --2- 172.21.15.76:0/3921768174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d280f63e0 0x7f5d280f6800 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f5d1c009fb0 tx=0x7f5d1c009b30).stop 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 -- 172.21.15.76:0/3921768174 shutdown_connections 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 --2- 172.21.15.76:0/3921768174 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d280f63e0 0x7f5d280f6800 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 -- 172.21.15.76:0/3921768174 >> 172.21.15.76:0/3921768174 conn(0x7f5d280f1b40 msgr2=0x7f5d280f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 -- 172.21.15.76:0/3921768174 shutdown_connections 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.777+0000 7f5d2cff7700 1 -- 172.21.15.76:0/3921768174 wait complete. 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d2cff7700 1 Processor -- start 2023-12-26T22:00:43.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d2cff7700 1 -- start start 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d2cff7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 0x7f5d28106b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d2cff7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d28107060 con 0x7f5d28106750 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 0x7f5d28106b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d277fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 0x7f5d28106b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44244/0 (socket says 172.21.15.76:44244) 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d277fe700 1 -- 172.21.15.76:0/2958885841 learned_addr learned my addr 172.21.15.76:0/2958885841 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.778+0000 7f5d277fe700 1 -- 172.21.15.76:0/2958885841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d1c005040 con 0x7f5d28106750 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.779+0000 7f5d277fe700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 0x7f5d28106b70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5d1c0134d0 tx=0x7f5d1c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.779+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d1c013ee0 con 0x7f5d28106750 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.779+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d28107260 con 0x7f5d28106750 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.779+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5d1c013680 con 0x7f5d28106750 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.779+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d1c01ee90 con 0x7f5d28106750 2023-12-26T22:00:43.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.779+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d28107db0 con 0x7f5d28106750 2023-12-26T22:00:43.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.780+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5d1c025060 con 0x7f5d28106750 2023-12-26T22:00:43.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.780+0000 7f5d0ffff700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5d10031040 0x7f5d10033500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:43.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.780+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d280591e0 con 0x7f5d28106750 2023-12-26T22:00:43.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.780+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5d1c042280 con 0x7f5d28106750 2023-12-26T22:00:43.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.780+0000 7f5d26ffd700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5d10031040 0x7f5d10033500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:43.780 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.781+0000 7f5d26ffd700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5d10031040 0x7f5d10033500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5d180096a0 tx=0x7f5d18006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:43.784 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.785+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5d1c01c520 con 0x7f5d28106750 2023-12-26T22:00:43.992 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.993+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5d28045bb0 con 0x7f5d28106750 2023-12-26T22:00:43.993 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.994+0000 7f5d0ffff700 1 -- 172.21.15.76:0/2958885841 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5d1c021110 con 0x7f5d28106750 2023-12-26T22:00:43.993 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:43.993 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:43.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5d10031040 msgr2=0x7f5d10033500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5d10031040 0x7f5d10033500 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5d180096a0 tx=0x7f5d18006b40).stop 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 msgr2=0x7f5d28106b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 0x7f5d28106b70 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f5d1c0134d0 tx=0x7f5d1c009b30).stop 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 shutdown_connections 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5d10031040 0x7f5d10033500 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 --2- 172.21.15.76:0/2958885841 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5d28106750 0x7f5d28106b70 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 >> 172.21.15.76:0/2958885841 conn(0x7f5d280f1b40 msgr2=0x7f5d280f26f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 shutdown_connections 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:43.995+0000 7f5d2cff7700 1 -- 172.21.15.76:0/2958885841 wait complete. 2023-12-26T22:00:43.995 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:45.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:43.361275+0000 mgr.smithi059.gwkiyc (mgr.14156) 121 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:45.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:43.994401+0000 mon.smithi059 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.76:0/2958885841' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:45.422 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:45.422 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:45.468424+0000 mon.smithi059 (mon.0) 190 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T22:00:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:45.468617+0000 mon.smithi059 (mon.0) 191 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 -- 172.21.15.76:0/414367690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 msgr2=0x7fbab00f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 --2- 172.21.15.76:0/414367690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab00f91c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7fba98009fb0 tx=0x7fba98009b30).stop 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 -- 172.21.15.76:0/414367690 shutdown_connections 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 --2- 172.21.15.76:0/414367690 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab00f91c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 -- 172.21.15.76:0/414367690 >> 172.21.15.76:0/414367690 conn(0x7fbab00f0be0 msgr2=0x7fbab00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 -- 172.21.15.76:0/414367690 shutdown_connections 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.663+0000 7fbab5088700 1 -- 172.21.15.76:0/414367690 wait complete. 2023-12-26T22:00:46.663 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbab5088700 1 Processor -- start 2023-12-26T22:00:46.664 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbab5088700 1 -- start start 2023-12-26T22:00:46.664 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbab5088700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:46.664 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbab5088700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbab0106c70 con 0x7fbab0060300 2023-12-26T22:00:46.665 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbaaed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:46.665 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbaaed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55156/0 (socket says 172.21.15.76:55156) 2023-12-26T22:00:46.665 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbaaed9d700 1 -- 172.21.15.76:0/3383536186 learned_addr learned my addr 172.21.15.76:0/3383536186 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.664+0000 7fbaaed9d700 1 -- 172.21.15.76:0/3383536186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fba98005040 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.665+0000 7fbaaed9d700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab0106780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fba98009f80 tx=0x7fba9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.665+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba98009320 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.665+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fba98013680 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.665+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbab0106e70 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.665+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fba9801eda0 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.665+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbab0107300 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.666+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fba9801e420 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.666+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbab0045bb0 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.666+0000 7fbaa77fe700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fba9c031090 0x7fba9c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.666+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fba98042360 con 0x7fbab0060300 2023-12-26T22:00:46.666 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.666+0000 7fbaae59c700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fba9c031090 0x7fba9c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:46.667 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.667+0000 7fbaae59c700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fba9c031090 0x7fba9c033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fbaa00096a0 tx=0x7fbaa0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:46.670 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.671+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fba9801d380 con 0x7fbab0060300 2023-12-26T22:00:46.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.874+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbab01081b0 con 0x7fbab0060300 2023-12-26T22:00:46.874 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.875+0000 7fbaa77fe700 1 -- 172.21.15.76:0/3383536186 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fba98004490 con 0x7fbab0060300 2023-12-26T22:00:46.874 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:46.874 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fba9c031090 msgr2=0x7fba9c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fba9c031090 0x7fba9c033550 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fbaa00096a0 tx=0x7fbaa0006b40).stop 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 msgr2=0x7fbab0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab0106780 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fba98009f80 tx=0x7fba9800b040).stop 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 shutdown_connections 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fba9c031090 0x7fba9c033550 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 --2- 172.21.15.76:0/3383536186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbab0060300 0x7fbab0106780 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 >> 172.21.15.76:0/3383536186 conn(0x7fbab00f0be0 msgr2=0x7fbab00f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:46.876 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 shutdown_connections 2023-12-26T22:00:46.877 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:46.877+0000 7fbab5088700 1 -- 172.21.15.76:0/3383536186 wait complete. 2023-12-26T22:00:46.877 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:47.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:45.361699+0000 mgr.smithi059.gwkiyc (mgr.14156) 122 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:48.271 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:48.272 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:48.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:47 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:46.875366+0000 mon.smithi059 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.76:0/3383536186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:47.362083+0000 mgr.smithi059.gwkiyc (mgr.14156) 123 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:49.518 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- 172.21.15.76:0/1962287519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 msgr2=0x7fc0400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:49.518 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 --2- 172.21.15.76:0/1962287519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0400f58a0 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fc030009fb0 tx=0x7fc030009b30).stop 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- 172.21.15.76:0/1962287519 shutdown_connections 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 --2- 172.21.15.76:0/1962287519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0400f58a0 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- 172.21.15.76:0/1962287519 >> 172.21.15.76:0/1962287519 conn(0x7fc0400f0be0 msgr2=0x7fc0400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- 172.21.15.76:0/1962287519 shutdown_connections 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- 172.21.15.76:0/1962287519 wait complete. 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 Processor -- start 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- start start 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0401094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.517+0000 7fc044fe2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc040060320 con 0x7fc0400f5480 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc03e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0401094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc03e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0401094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55172/0 (socket says 172.21.15.76:55172) 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc03e59c700 1 -- 172.21.15.76:0/3793468497 learned_addr learned my addr 172.21.15.76:0/3793468497 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc03e59c700 1 -- 172.21.15.76:0/3793468497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc030005040 con 0x7fc0400f5480 2023-12-26T22:00:49.519 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc03e59c700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0401094f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc0300089f0 tx=0x7fc03000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:49.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc030009320 con 0x7fc0400f5480 2023-12-26T22:00:49.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc030013ec0 con 0x7fc0400f5480 2023-12-26T22:00:49.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc040060520 con 0x7fc0400f5480 2023-12-26T22:00:49.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc030014d30 con 0x7fc0400f5480 2023-12-26T22:00:49.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.518+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc040109ba0 con 0x7fc0400f5480 2023-12-26T22:00:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.519+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc0300095f0 con 0x7fc0400f5480 2023-12-26T22:00:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.519+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc040045bb0 con 0x7fc0400f5480 2023-12-26T22:00:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.519+0000 7fc02effd700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc028031090 0x7fc028033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.519+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc0300424a0 con 0x7fc0400f5480 2023-12-26T22:00:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.519+0000 7fc03dd9b700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc028031090 0x7fc028033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:49.521 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.520+0000 7fc03dd9b700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc028031090 0x7fc028033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc0340096a0 tx=0x7fc034006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:49.524 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.524+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc03001d7f0 con 0x7fc0400f5480 2023-12-26T22:00:49.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.728+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc0400faf70 con 0x7fc0400f5480 2023-12-26T22:00:49.729 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.729+0000 7fc02effd700 1 -- 172.21.15.76:0/3793468497 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc03001d7f0 con 0x7fc0400f5480 2023-12-26T22:00:49.729 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:49.729 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:49.730 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc028031090 msgr2=0x7fc028033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc028031090 0x7fc028033550 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fc0340096a0 tx=0x7fc034006b40).stop 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 msgr2=0x7fc0401094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0401094f0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fc0300089f0 tx=0x7fc03000b040).stop 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 shutdown_connections 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc028031090 0x7fc028033550 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 --2- 172.21.15.76:0/3793468497 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0400f5480 0x7fc0401094f0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 >> 172.21.15.76:0/3793468497 conn(0x7fc0400f0be0 msgr2=0x7fc0400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 shutdown_connections 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:49.730+0000 7fc044fe2700 1 -- 172.21.15.76:0/3793468497 wait complete. 2023-12-26T22:00:49.731 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:50.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:49 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:49.730425+0000 mon.smithi059 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.76:0/3793468497' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:51.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:51.137 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:49.362399+0000 mgr.smithi059.gwkiyc (mgr.14156) 124 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:52.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 -- 172.21.15.76:0/3950619598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 msgr2=0x7fc5200f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 --2- 172.21.15.76:0/3950619598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5200f69b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7fc518009960 tx=0x7fc518004d10).stop 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 -- 172.21.15.76:0/3950619598 shutdown_connections 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 --2- 172.21.15.76:0/3950619598 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5200f69b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 -- 172.21.15.76:0/3950619598 >> 172.21.15.76:0/3950619598 conn(0x7fc5200f1cf0 msgr2=0x7fc5200f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 -- 172.21.15.76:0/3950619598 shutdown_connections 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 -- 172.21.15.76:0/3950619598 wait complete. 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 Processor -- start 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.352+0000 7fc527249700 1 -- start start 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc527249700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5201089b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc527249700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc520108ea0 con 0x7fc5200f6590 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc524fe5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5201089b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:52.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc524fe5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5201089b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55190/0 (socket says 172.21.15.76:55190) 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc524fe5700 1 -- 172.21.15.76:0/2516765096 learned_addr learned my addr 172.21.15.76:0/2516765096 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc524fe5700 1 -- 172.21.15.76:0/2516765096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc518009640 con 0x7fc5200f6590 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc524fe5700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5201089b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc518009610 tx=0x7fc518004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc518018070 con 0x7fc5200f6590 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5201090a0 con 0x7fc5200f6590 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc51800b7e0 con 0x7fc5200f6590 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.353+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc51801d420 con 0x7fc5200f6590 2023-12-26T22:00:52.355 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.354+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc520109530 con 0x7fc5200f6590 2023-12-26T22:00:52.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.354+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc518024060 con 0x7fc5200f6590 2023-12-26T22:00:52.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.354+0000 7fc51d7fa700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc508031090 0x7fc508033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:52.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.354+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc518042690 con 0x7fc5200f6590 2023-12-26T22:00:52.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.354+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc520045bb0 con 0x7fc5200f6590 2023-12-26T22:00:52.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.355+0000 7fc51ffff700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc508031090 0x7fc508033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:52.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.355+0000 7fc51ffff700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc508031090 0x7fc508033550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc5140096a0 tx=0x7fc514006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:52.359 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.360+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc518011500 con 0x7fc5200f6590 2023-12-26T22:00:52.567 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.567+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc520102420 con 0x7fc5200f6590 2023-12-26T22:00:52.568 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.568+0000 7fc51d7fa700 1 -- 172.21.15.76:0/2516765096 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc518011320 con 0x7fc5200f6590 2023-12-26T22:00:52.568 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:52.568 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.569+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc508031090 msgr2=0x7fc508033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.569+0000 7fc527249700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc508031090 0x7fc508033550 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7fc5140096a0 tx=0x7fc514006b40).stop 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.569+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 msgr2=0x7fc5201089b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.569+0000 7fc527249700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5201089b0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7fc518009610 tx=0x7fc518004d10).stop 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.570+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 shutdown_connections 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.570+0000 7fc527249700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc508031090 0x7fc508033550 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.570+0000 7fc527249700 1 --2- 172.21.15.76:0/2516765096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc5200f6590 0x7fc5201089b0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.570+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 >> 172.21.15.76:0/2516765096 conn(0x7fc5200f1cf0 msgr2=0x7fc5200f40f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.570+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 shutdown_connections 2023-12-26T22:00:52.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:52.570+0000 7fc527249700 1 -- 172.21.15.76:0/2516765096 wait complete. 2023-12-26T22:00:52.571 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:51.362735+0000 mgr.smithi059.gwkiyc (mgr.14156) 125 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:52.569381+0000 mon.smithi059 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.76:0/2516765096' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:53.958 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:53.959 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:55.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 -- 172.21.15.76:0/497069788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 msgr2=0x7f95d80f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 --2- 172.21.15.76:0/497069788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d80f69b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f95cc004760 tx=0x7f95cc009ad0).stop 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 -- 172.21.15.76:0/497069788 shutdown_connections 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 --2- 172.21.15.76:0/497069788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d80f69b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 -- 172.21.15.76:0/497069788 >> 172.21.15.76:0/497069788 conn(0x7f95d80f1cd0 msgr2=0x7f95d80f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 -- 172.21.15.76:0/497069788 shutdown_connections 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 -- 172.21.15.76:0/497069788 wait complete. 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 Processor -- start 2023-12-26T22:00:55.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.174+0000 7f95deea7700 1 -- start start 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95deea7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95deea7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95d8069f00 con 0x7f95d80f6590 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95dcc43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95dcc43700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:55206/0 (socket says 172.21.15.76:55206) 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95dcc43700 1 -- 172.21.15.76:0/1726461355 learned_addr learned my addr 172.21.15.76:0/1726461355 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95dcc43700 1 -- 172.21.15.76:0/1726461355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95cc009d70 con 0x7f95d80f6590 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95dcc43700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d806ba80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f95cc004170 tx=0x7f95cc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:55.176 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95cc00e040 con 0x7f95d80f6590 2023-12-26T22:00:55.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.175+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f95cc008ef0 con 0x7f95d80f6590 2023-12-26T22:00:55.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.176+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95d806a0a0 con 0x7f95d80f6590 2023-12-26T22:00:55.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.176+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95cc014c90 con 0x7f95d80f6590 2023-12-26T22:00:55.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.176+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95d806a530 con 0x7f95d80f6590 2023-12-26T22:00:55.177 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.176+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f95cc0283f0 con 0x7f95d80f6590 2023-12-26T22:00:55.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.177+0000 7f95d57fa700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f95c80310e0 0x7f95c80335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:55.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.177+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95d80591e0 con 0x7f95d80f6590 2023-12-26T22:00:55.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.177+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f95cc0422e0 con 0x7f95d80f6590 2023-12-26T22:00:55.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.177+0000 7f95d7fff700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f95c80310e0 0x7f95c80335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:55.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.178+0000 7f95d7fff700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f95c80310e0 0x7f95c80335a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f95c40096a0 tx=0x7f95c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:55.182 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.182+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f95cc0266c0 con 0x7f95d80f6590 2023-12-26T22:00:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:53.363124+0000 mgr.smithi059.gwkiyc (mgr.14156) 126 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:55.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.387+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95d80fc080 con 0x7f95d80f6590 2023-12-26T22:00:55.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.388+0000 7f95d57fa700 1 -- 172.21.15.76:0/1726461355 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f95cc01d9c0 con 0x7f95d80f6590 2023-12-26T22:00:55.388 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:55.388 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:55.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f95c80310e0 msgr2=0x7f95c80335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f95c80310e0 0x7f95c80335a0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f95c40096a0 tx=0x7f95c4006b40).stop 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 msgr2=0x7f95d806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d806ba80 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f95cc004170 tx=0x7f95cc009ad0).stop 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 shutdown_connections 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f95c80310e0 0x7f95c80335a0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 --2- 172.21.15.76:0/1726461355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f95d80f6590 0x7f95d806ba80 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 >> 172.21.15.76:0/1726461355 conn(0x7f95d80f1cd0 msgr2=0x7f95d80f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 shutdown_connections 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:55.389+0000 7f95deea7700 1 -- 172.21.15.76:0/1726461355 wait complete. 2023-12-26T22:00:55.390 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:56.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:55.389263+0000 mon.smithi059 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.76:0/1726461355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:56.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:56.776 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:00:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:55.363435+0000 mgr.smithi059.gwkiyc (mgr.14156) 127 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:58.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:57.999+0000 7fd50fe25700 1 -- 172.21.15.76:0/3061941814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 msgr2=0x7fd5080f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:58.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:57.999+0000 7fd50fe25700 1 --2- 172.21.15.76:0/3061941814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5080f58a0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7fd4fc009fb0 tx=0x7fd4fc009b30).stop 2023-12-26T22:00:58.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:57.999+0000 7fd50fe25700 1 -- 172.21.15.76:0/3061941814 shutdown_connections 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:57.999+0000 7fd50fe25700 1 --2- 172.21.15.76:0/3061941814 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5080f58a0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:57.999+0000 7fd50fe25700 1 -- 172.21.15.76:0/3061941814 >> 172.21.15.76:0/3061941814 conn(0x7fd5080f0be0 msgr2=0x7fd5080f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50fe25700 1 -- 172.21.15.76:0/3061941814 shutdown_connections 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50fe25700 1 -- 172.21.15.76:0/3061941814 wait complete. 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50fe25700 1 Processor -- start 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50fe25700 1 -- start start 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50fe25700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5081056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50fe25700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd508105be0 con 0x7fd5080f5480 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50dbc1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5081056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50dbc1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5081056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:43452/0 (socket says 172.21.15.76:43452) 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.000+0000 7fd50dbc1700 1 -- 172.21.15.76:0/711761973 learned_addr learned my addr 172.21.15.76:0/711761973 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd50dbc1700 1 -- 172.21.15.76:0/711761973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4fc005040 con 0x7fd5080f5480 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd50dbc1700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5081056f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fd4fc0089f0 tx=0x7fd4fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:58.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4fc009320 con 0x7fd5080f5480 2023-12-26T22:00:58.003 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd508105de0 con 0x7fd5080f5480 2023-12-26T22:00:58.003 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd4fc013ec0 con 0x7fd5080f5480 2023-12-26T22:00:58.003 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4fc014d30 con 0x7fd5080f5480 2023-12-26T22:00:58.003 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.001+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd508106270 con 0x7fd5080f5480 2023-12-26T22:00:58.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.002+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd4fc0263a0 con 0x7fd5080f5480 2023-12-26T22:00:58.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.002+0000 7fd4fa7fc700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd4f4031090 0x7fd4f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:00:58.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.002+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd4fc0423b0 con 0x7fd5080f5480 2023-12-26T22:00:58.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.002+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd508045bb0 con 0x7fd5080f5480 2023-12-26T22:00:58.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.002+0000 7fd50d3c0700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd4f4031090 0x7fd4f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:00:58.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.003+0000 7fd50d3c0700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd4f4031090 0x7fd4f4033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd5040096a0 tx=0x7fd504006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:00:58.007 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.007+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd4fc0266d0 con 0x7fd5080f5480 2023-12-26T22:00:58.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.213+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd5080fb060 con 0x7fd5080f5480 2023-12-26T22:00:58.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.213+0000 7fd4fa7fc700 1 -- 172.21.15.76:0/711761973 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd4fc012330 con 0x7fd5080f5480 2023-12-26T22:00:58.213 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:00:58.214 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd4f4031090 msgr2=0x7fd4f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd4f4031090 0x7fd4f4033550 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fd5040096a0 tx=0x7fd504006b40).stop 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 msgr2=0x7fd5081056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5081056f0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7fd4fc0089f0 tx=0x7fd4fc00b040).stop 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 shutdown_connections 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd4f4031090 0x7fd4f4033550 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:58.215 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 --2- 172.21.15.76:0/711761973 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd5080f5480 0x7fd5081056f0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:00:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 >> 172.21.15.76:0/711761973 conn(0x7fd5080f0be0 msgr2=0x7fd5080f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:00:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 shutdown_connections 2023-12-26T22:00:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:00:58.215+0000 7fd50fe25700 1 -- 172.21.15.76:0/711761973 wait complete. 2023-12-26T22:00:58.216 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:00:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:57.363685+0000 mgr.smithi059.gwkiyc (mgr.14156) 128 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:00:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:00:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:00:58.214912+0000 mon.smithi059 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.76:0/711761973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:00:59.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:00:59.624 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 -- 172.21.15.76:0/745196528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 msgr2=0x7f15c40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 --2- 172.21.15.76:0/745196528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c40f58a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f15ac009fb0 tx=0x7f15ac009b30).stop 2023-12-26T22:01:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 -- 172.21.15.76:0/745196528 shutdown_connections 2023-12-26T22:01:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 --2- 172.21.15.76:0/745196528 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c40f58a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:00.863 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 -- 172.21.15.76:0/745196528 >> 172.21.15.76:0/745196528 conn(0x7f15c40f0be0 msgr2=0x7f15c40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 -- 172.21.15.76:0/745196528 shutdown_connections 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 -- 172.21.15.76:0/745196528 wait complete. 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.861+0000 7f15ca2f9700 1 Processor -- start 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15ca2f9700 1 -- start start 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15ca2f9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c4109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15ca2f9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c4060300 con 0x7f15c40f5480 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15c3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c4109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15c3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c4109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:43462/0 (socket says 172.21.15.76:43462) 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15c3fff700 1 -- 172.21.15.76:0/2275442735 learned_addr learned my addr 172.21.15.76:0/2275442735 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15c3fff700 1 -- 172.21.15.76:0/2275442735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15ac005040 con 0x7f15c40f5480 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.862+0000 7f15c3fff700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c4109510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f15ac0089f0 tx=0x7f15ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.863+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15ac009320 con 0x7f15c40f5480 2023-12-26T22:01:00.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.863+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c4060500 con 0x7f15c40f5480 2023-12-26T22:01:00.865 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.863+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f15ac013ec0 con 0x7f15c40f5480 2023-12-26T22:01:00.865 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.863+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15ac014d30 con 0x7f15c40f5480 2023-12-26T22:01:00.865 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.863+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c4109bc0 con 0x7f15c40f5480 2023-12-26T22:01:00.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.863+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f15ac0263a0 con 0x7f15c40f5480 2023-12-26T22:01:00.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.864+0000 7f15c0ff9700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f15b0031090 0x7f15b0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:00.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.864+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f15ac0424b0 con 0x7f15c40f5480 2023-12-26T22:01:00.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.864+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c40591e0 con 0x7f15c40f5480 2023-12-26T22:01:00.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.864+0000 7f15c37fe700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f15b0031090 0x7f15b0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:00.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.864+0000 7f15c37fe700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f15b0031090 0x7f15b0033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f15b40096a0 tx=0x7f15b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:00.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:00.868+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f15ac0266d0 con 0x7f15c40f5480 2023-12-26T22:01:01.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.075+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15c4045bb0 con 0x7f15c40f5480 2023-12-26T22:01:01.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.076+0000 7f15c0ff9700 1 -- 172.21.15.76:0/2275442735 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f15ac012330 con 0x7f15c40f5480 2023-12-26T22:01:01.076 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:01.076 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:01.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f15b0031090 msgr2=0x7f15b0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:01.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f15b0031090 0x7f15b0033550 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f15b40096a0 tx=0x7f15b4006b40).stop 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 msgr2=0x7f15c4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c4109510 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f15ac0089f0 tx=0x7f15ac00b040).stop 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 shutdown_connections 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f15b0031090 0x7f15b0033550 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 --2- 172.21.15.76:0/2275442735 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f15c40f5480 0x7f15c4109510 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 >> 172.21.15.76:0/2275442735 conn(0x7f15c40f0be0 msgr2=0x7f15c40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 shutdown_connections 2023-12-26T22:01:01.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:01.077+0000 7f15ca2f9700 1 -- 172.21.15.76:0/2275442735 wait complete. 2023-12-26T22:01:01.079 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:01.096 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:00:59.363977+0000 mgr.smithi059.gwkiyc (mgr.14156) 129 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:02.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:01.077180+0000 mon.smithi059 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.76:0/2275442735' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:02.474 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:02.475 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:01.364353+0000 mgr.smithi059.gwkiyc (mgr.14156) 130 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:03.634 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 -- 172.21.15.76:0/2399028473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 msgr2=0x7efc70069f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 --2- 172.21.15.76:0/2399028473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc70069f60 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7efc60009fb0 tx=0x7efc60009b30).stop 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 -- 172.21.15.76:0/2399028473 shutdown_connections 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 --2- 172.21.15.76:0/2399028473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc70069f60 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 -- 172.21.15.76:0/2399028473 >> 172.21.15.76:0/2399028473 conn(0x7efc700f1b40 msgr2=0x7efc700f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 -- 172.21.15.76:0/2399028473 shutdown_connections 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.633+0000 7efc74df3700 1 -- 172.21.15.76:0/2399028473 wait complete. 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc74df3700 1 Processor -- start 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc74df3700 1 -- start start 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc74df3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc701087e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc74df3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc70108cd0 con 0x7efc7006ba60 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc701087e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc6f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc701087e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:43484/0 (socket says 172.21.15.76:43484) 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc6f7fe700 1 -- 172.21.15.76:0/63152352 learned_addr learned my addr 172.21.15.76:0/63152352 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:03.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.634+0000 7efc6f7fe700 1 -- 172.21.15.76:0/63152352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60005040 con 0x7efc7006ba60 2023-12-26T22:01:03.636 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.635+0000 7efc6f7fe700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc701087e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7efc60009f80 tx=0x7efc6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.635+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc60009320 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.635+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc70108ed0 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.635+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efc60013680 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.635+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc6001eda0 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.635+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc70109360 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.636+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efc60014420 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.636+0000 7efc57fff700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc58031090 0x7efc58033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.636+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efc60042360 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.636+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc70045bb0 con 0x7efc7006ba60 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.636+0000 7efc6effd700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc58031090 0x7efc58033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:03.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.636+0000 7efc6effd700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc58031090 0x7efc58033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7efc640096a0 tx=0x7efc64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:03.640 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.641+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efc60014700 con 0x7efc7006ba60 2023-12-26T22:01:03.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.854+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efc7010a160 con 0x7efc7006ba60 2023-12-26T22:01:03.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.855+0000 7efc57fff700 1 -- 172.21.15.76:0/63152352 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efc60026350 con 0x7efc7006ba60 2023-12-26T22:01:03.855 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:03.856 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.856+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc58031090 msgr2=0x7efc58033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.856+0000 7efc74df3700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc58031090 0x7efc58033550 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7efc640096a0 tx=0x7efc64006b40).stop 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.856+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 msgr2=0x7efc701087e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.856+0000 7efc74df3700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc701087e0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7efc60009f80 tx=0x7efc6000b040).stop 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.857+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 shutdown_connections 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.857+0000 7efc74df3700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efc58031090 0x7efc58033550 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.857+0000 7efc74df3700 1 --2- 172.21.15.76:0/63152352 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efc7006ba60 0x7efc701087e0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.857+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 >> 172.21.15.76:0/63152352 conn(0x7efc700f1b40 msgr2=0x7efc700f2610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.857+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 shutdown_connections 2023-12-26T22:01:03.857 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:03.857+0000 7efc74df3700 1 -- 172.21.15.76:0/63152352 wait complete. 2023-12-26T22:01:03.858 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:05.239 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:05.239 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:03.364711+0000 mgr.smithi059.gwkiyc (mgr.14156) 131 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:03.856659+0000 mon.smithi059 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.76:0/63152352' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.418+0000 7fe10d2aa700 1 -- 172.21.15.76:0/833303344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 msgr2=0x7fe1080f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.418+0000 7fe10d2aa700 1 --2- 172.21.15.76:0/833303344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1080f4c70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fe0fc009fb0 tx=0x7fe0fc009b30).stop 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.418+0000 7fe10d2aa700 1 -- 172.21.15.76:0/833303344 shutdown_connections 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.418+0000 7fe10d2aa700 1 --2- 172.21.15.76:0/833303344 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1080f4c70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.418+0000 7fe10d2aa700 1 -- 172.21.15.76:0/833303344 >> 172.21.15.76:0/833303344 conn(0x7fe1080f0390 msgr2=0x7fe1080f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.418+0000 7fe10d2aa700 1 -- 172.21.15.76:0/833303344 shutdown_connections 2023-12-26T22:01:06.419 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.419+0000 7fe10d2aa700 1 -- 172.21.15.76:0/833303344 wait complete. 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.419+0000 7fe10d2aa700 1 Processor -- start 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.419+0000 7fe10d2aa700 1 -- start start 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.419+0000 7fe10d2aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1081067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.419+0000 7fe10d2aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe108106c90 con 0x7fe1080f4850 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe106ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1081067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe106ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1081067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52166/0 (socket says 172.21.15.76:52166) 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe106ffd700 1 -- 172.21.15.76:0/896918160 learned_addr learned my addr 172.21.15.76:0/896918160 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe106ffd700 1 -- 172.21.15.76:0/896918160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0fc005040 con 0x7fe1080f4850 2023-12-26T22:01:06.421 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe106ffd700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1081067a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fe0fc0089f0 tx=0x7fe0fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:06.422 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0fc009320 con 0x7fe1080f4850 2023-12-26T22:01:06.422 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe108106e90 con 0x7fe1080f4850 2023-12-26T22:01:06.422 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe0fc013ec0 con 0x7fe1080f4850 2023-12-26T22:01:06.422 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe0fc014d30 con 0x7fe1080f4850 2023-12-26T22:01:06.422 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.420+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe108107320 con 0x7fe1080f4850 2023-12-26T22:01:06.423 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.421+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe0fc0095f0 con 0x7fe1080f4850 2023-12-26T22:01:06.423 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.421+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe108045bb0 con 0x7fe1080f4850 2023-12-26T22:01:06.423 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.421+0000 7fe0ef7fe700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0f00310e0 0x7fe0f00335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:06.423 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.421+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe0fc0424a0 con 0x7fe1080f4850 2023-12-26T22:01:06.423 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.422+0000 7fe1067fc700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0f00310e0 0x7fe0f00335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:06.423 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.422+0000 7fe1067fc700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0f00310e0 0x7fe0f00335a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fe0f80096a0 tx=0x7fe0f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:06.426 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.427+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe0fc01d980 con 0x7fe1080f4850 2023-12-26T22:01:06.634 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.634+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe1081081d0 con 0x7fe1080f4850 2023-12-26T22:01:06.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.636+0000 7fe0ef7fe700 1 -- 172.21.15.76:0/896918160 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe0fc01d380 con 0x7fe1080f4850 2023-12-26T22:01:06.635 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:06.636 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.636+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0f00310e0 msgr2=0x7fe0f00335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.636+0000 7fe10d2aa700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0f00310e0 0x7fe0f00335a0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fe0f80096a0 tx=0x7fe0f8006b40).stop 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.636+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 msgr2=0x7fe1081067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.636+0000 7fe10d2aa700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1081067a0 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7fe0fc0089f0 tx=0x7fe0fc00b040).stop 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.637+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 shutdown_connections 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.637+0000 7fe10d2aa700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0f00310e0 0x7fe0f00335a0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.637+0000 7fe10d2aa700 1 --2- 172.21.15.76:0/896918160 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe1080f4850 0x7fe1081067a0 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.637+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 >> 172.21.15.76:0/896918160 conn(0x7fe1080f0390 msgr2=0x7fe1080f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:06.637 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.637+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 shutdown_connections 2023-12-26T22:01:06.638 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:06.637+0000 7fe10d2aa700 1 -- 172.21.15.76:0/896918160 wait complete. 2023-12-26T22:01:06.638 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:05.365091+0000 mgr.smithi059.gwkiyc (mgr.14156) 132 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:06.636490+0000 mon.smithi059 (mon.0) 199 : audit [DBG] from='client.? 172.21.15.76:0/896918160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:08.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:08.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 -- 172.21.15.76:0/3862696568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 msgr2=0x7f2ccc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 --2- 172.21.15.76:0/3862696568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc0f69b0 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2cbc004760 tx=0x7f2cbc009ad0).stop 2023-12-26T22:01:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 -- 172.21.15.76:0/3862696568 shutdown_connections 2023-12-26T22:01:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 --2- 172.21.15.76:0/3862696568 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc0f69b0 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:09.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 -- 172.21.15.76:0/3862696568 >> 172.21.15.76:0/3862696568 conn(0x7f2ccc0f1cd0 msgr2=0x7f2ccc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:09.226 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 -- 172.21.15.76:0/3862696568 shutdown_connections 2023-12-26T22:01:09.226 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.225+0000 7f2cd2921700 1 -- 172.21.15.76:0/3862696568 wait complete. 2023-12-26T22:01:09.226 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.226+0000 7f2cd2921700 1 Processor -- start 2023-12-26T22:01:09.226 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.226+0000 7f2cd2921700 1 -- start start 2023-12-26T22:01:09.226 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.226+0000 7f2cd2921700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:09.226 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.226+0000 7f2cd2921700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ccc069f00 con 0x7f2ccc0f6590 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2ccbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2ccbfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52188/0 (socket says 172.21.15.76:52188) 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2ccbfff700 1 -- 172.21.15.76:0/3828679423 learned_addr learned my addr 172.21.15.76:0/3828679423 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2ccbfff700 1 -- 172.21.15.76:0/3828679423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cbc009d70 con 0x7f2ccc0f6590 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2ccbfff700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc06ba80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f2cbc004170 tx=0x7f2cbc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cbc00e040 con 0x7f2ccc0f6590 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ccc06a0a0 con 0x7f2ccc0f6590 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2cbc008ef0 con 0x7f2ccc0f6590 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2cbc014c90 con 0x7f2ccc0f6590 2023-12-26T22:01:09.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.227+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ccc06a530 con 0x7f2ccc0f6590 2023-12-26T22:01:09.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.228+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2cbc0283f0 con 0x7f2ccc0f6590 2023-12-26T22:01:09.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.228+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ccc0591e0 con 0x7f2ccc0f6590 2023-12-26T22:01:09.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.228+0000 7f2cc8ff9700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cb8031090 0x7f2cb8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:09.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.228+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2cbc042240 con 0x7f2ccc0f6590 2023-12-26T22:01:09.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.229+0000 7f2ccb7fe700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cb8031090 0x7f2cb8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:09.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.229+0000 7f2ccb7fe700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cb8031090 0x7f2cb8033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2cb40096a0 tx=0x7f2cb4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:09.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.233+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2cbc0266c0 con 0x7f2ccc0f6590 2023-12-26T22:01:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:07.365536+0000 mgr.smithi059.gwkiyc (mgr.14156) 133 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:09.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.438+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2ccc0fc080 con 0x7f2ccc0f6590 2023-12-26T22:01:09.437 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.439+0000 7f2cc8ff9700 1 -- 172.21.15.76:0/3828679423 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2cbc026300 con 0x7f2ccc0f6590 2023-12-26T22:01:09.438 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:09.438 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:09.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cb8031090 msgr2=0x7f2cb8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:09.439 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cb8031090 0x7f2cb8033550 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2cb40096a0 tx=0x7f2cb4006b40).stop 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 msgr2=0x7f2ccc06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc06ba80 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f2cbc004170 tx=0x7f2cbc009ad0).stop 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 shutdown_connections 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cb8031090 0x7f2cb8033550 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 --2- 172.21.15.76:0/3828679423 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2ccc0f6590 0x7f2ccc06ba80 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 >> 172.21.15.76:0/3828679423 conn(0x7f2ccc0f1cd0 msgr2=0x7f2ccc0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 shutdown_connections 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:09.440+0000 7f2cd2921700 1 -- 172.21.15.76:0/3828679423 wait complete. 2023-12-26T22:01:09.440 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:10.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:09 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:09.439242+0000 mon.smithi059 (mon.0) 200 : audit [DBG] from='client.? 172.21.15.76:0/3828679423' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:10.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:10.826 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:09.365890+0000 mgr.smithi059.gwkiyc (mgr.14156) 134 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 -- 172.21.15.76:0/2678831788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 msgr2=0x7f64940f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:12.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 --2- 172.21.15.76:0/2678831788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64940f58d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f6488009fb0 tx=0x7f6488009b30).stop 2023-12-26T22:01:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 -- 172.21.15.76:0/2678831788 shutdown_connections 2023-12-26T22:01:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 --2- 172.21.15.76:0/2678831788 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64940f58d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 -- 172.21.15.76:0/2678831788 >> 172.21.15.76:0/2678831788 conn(0x7f64940f0c10 msgr2=0x7f64940f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 -- 172.21.15.76:0/2678831788 shutdown_connections 2023-12-26T22:01:12.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.041+0000 7f649993b700 1 -- 172.21.15.76:0/2678831788 wait complete. 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f649993b700 1 Processor -- start 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f649993b700 1 -- start start 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f649993b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64941056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f649993b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6494105be0 con 0x7f64940f54b0 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f6492ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64941056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f6492ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64941056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52206/0 (socket says 172.21.15.76:52206) 2023-12-26T22:01:12.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.042+0000 7f6492ffd700 1 -- 172.21.15.76:0/4174616111 learned_addr learned my addr 172.21.15.76:0/4174616111 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f6492ffd700 1 -- 172.21.15.76:0/4174616111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6488005040 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f6492ffd700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64941056f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f64880089f0 tx=0x7f648800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6488009320 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6494105de0 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6488013ec0 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6488014d30 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.043+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6494106270 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.044+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6488014420 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.044+0000 7f647b7fe700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f647c031090 0x7f647c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.044+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f64880423e0 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.044+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6494045bb0 con 0x7f64940f54b0 2023-12-26T22:01:12.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.044+0000 7f64927fc700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f647c031090 0x7f647c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:12.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.045+0000 7f64927fc700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f647c031090 0x7f647c033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f64840096a0 tx=0x7f6484006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:12.048 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.049+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6488014750 con 0x7f64940f54b0 2023-12-26T22:01:12.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.256+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6494107070 con 0x7f64940f54b0 2023-12-26T22:01:12.255 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.257+0000 7f647b7fe700 1 -- 172.21.15.76:0/4174616111 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f648801d7f0 con 0x7f64940f54b0 2023-12-26T22:01:12.256 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:12.256 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f647c031090 msgr2=0x7f647c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f647c031090 0x7f647c033550 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f64840096a0 tx=0x7f6484006b40).stop 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 msgr2=0x7f64941056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64941056f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f64880089f0 tx=0x7f648800b040).stop 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 shutdown_connections 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f647c031090 0x7f647c033550 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 --2- 172.21.15.76:0/4174616111 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f64940f54b0 0x7f64941056f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 >> 172.21.15.76:0/4174616111 conn(0x7f64940f0c10 msgr2=0x7f64940f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:12.258 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 shutdown_connections 2023-12-26T22:01:12.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:12.258+0000 7f649993b700 1 -- 172.21.15.76:0/4174616111 wait complete. 2023-12-26T22:01:12.259 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:13.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:11.366256+0000 mgr.smithi059.gwkiyc (mgr.14156) 135 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:13.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:12.257179+0000 mon.smithi059 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.76:0/4174616111' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:13.645 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:13.645 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 -- 172.21.15.76:0/3557981628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 msgr2=0x7f40dc0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 --2- 172.21.15.76:0/3557981628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc0f9160 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f40d0009fb0 tx=0x7f40d0009b30).stop 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 -- 172.21.15.76:0/3557981628 shutdown_connections 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 --2- 172.21.15.76:0/3557981628 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc0f9160 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 -- 172.21.15.76:0/3557981628 >> 172.21.15.76:0/3557981628 conn(0x7f40dc0f0be0 msgr2=0x7f40dc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 -- 172.21.15.76:0/3557981628 shutdown_connections 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 -- 172.21.15.76:0/3557981628 wait complete. 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 Processor -- start 2023-12-26T22:01:14.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.896+0000 7f40e2ff5700 1 -- start start 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e2ff5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e2ff5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40dc106c40 con 0x7f40dc060360 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e0d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e0d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52230/0 (socket says 172.21.15.76:52230) 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e0d91700 1 -- 172.21.15.76:0/1539759485 learned_addr learned my addr 172.21.15.76:0/1539759485 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e0d91700 1 -- 172.21.15.76:0/1539759485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40d0005040 con 0x7f40dc060360 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e0d91700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc106750 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f40d0009f80 tx=0x7f40d000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d0009320 con 0x7f40dc060360 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f40d0013680 con 0x7f40dc060360 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.897+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40dc106e40 con 0x7f40dc060360 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.898+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40d001eda0 con 0x7f40dc060360 2023-12-26T22:01:14.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.898+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40dc1072d0 con 0x7f40dc060360 2023-12-26T22:01:14.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.898+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f40d001e420 con 0x7f40dc060360 2023-12-26T22:01:14.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.899+0000 7f40d97fa700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f40cc031130 0x7f40cc0335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:14.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.899+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40dc045bb0 con 0x7f40dc060360 2023-12-26T22:01:14.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.899+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f40d00423f0 con 0x7f40dc060360 2023-12-26T22:01:14.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.899+0000 7f40dbfff700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f40cc031130 0x7f40cc0335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:14.899 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.899+0000 7f40dbfff700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f40cc031130 0x7f40cc0335f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f40c80096a0 tx=0x7f40c8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:14.903 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:14.903+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f40d001e6b0 con 0x7f40dc060360 2023-12-26T22:01:15.109 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.110+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40dc1080d0 con 0x7f40dc060360 2023-12-26T22:01:15.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.111+0000 7f40d97fa700 1 -- 172.21.15.76:0/1539759485 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f40d001cb10 con 0x7f40dc060360 2023-12-26T22:01:15.110 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:15.110 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f40cc031130 msgr2=0x7f40cc0335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f40cc031130 0x7f40cc0335f0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f40c80096a0 tx=0x7f40c8006b40).stop 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 msgr2=0x7f40dc106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc106750 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f40d0009f80 tx=0x7f40d000b040).stop 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 shutdown_connections 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f40cc031130 0x7f40cc0335f0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:15.112 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 --2- 172.21.15.76:0/1539759485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f40dc060360 0x7f40dc106750 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:15.113 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 >> 172.21.15.76:0/1539759485 conn(0x7f40dc0f0be0 msgr2=0x7f40dc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:15.113 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 shutdown_connections 2023-12-26T22:01:15.113 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:15.113+0000 7f40e2ff5700 1 -- 172.21.15.76:0/1539759485 wait complete. 2023-12-26T22:01:15.113 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:13.366515+0000 mgr.smithi059.gwkiyc (mgr.14156) 136 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:16.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:15 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:15.111627+0000 mon.smithi059 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.76:0/1539759485' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:16.527 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:16.527 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:17.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:15.366856+0000 mgr.smithi059.gwkiyc (mgr.14156) 137 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:17.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 -- 172.21.15.76:0/2685615884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 msgr2=0x7fe7040f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:17.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 --2- 172.21.15.76:0/2685615884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7040f4c70 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fe6f8009fb0 tx=0x7fe6f8009b30).stop 2023-12-26T22:01:17.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 -- 172.21.15.76:0/2685615884 shutdown_connections 2023-12-26T22:01:17.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 --2- 172.21.15.76:0/2685615884 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7040f4c70 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:17.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 -- 172.21.15.76:0/2685615884 >> 172.21.15.76:0/2685615884 conn(0x7fe7040f0390 msgr2=0x7fe7040f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:17.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 -- 172.21.15.76:0/2685615884 shutdown_connections 2023-12-26T22:01:17.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.723+0000 7fe7093a9700 1 -- 172.21.15.76:0/2685615884 wait complete. 2023-12-26T22:01:17.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe7093a9700 1 Processor -- start 2023-12-26T22:01:17.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe7093a9700 1 -- start start 2023-12-26T22:01:17.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe7093a9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7041067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:17.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe7093a9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe704106c90 con 0x7fe7040f4850 2023-12-26T22:01:17.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe702ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7041067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe702ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7041067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:42758/0 (socket says 172.21.15.76:42758) 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe702ffd700 1 -- 172.21.15.76:0/2302783847 learned_addr learned my addr 172.21.15.76:0/2302783847 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.724+0000 7fe702ffd700 1 -- 172.21.15.76:0/2302783847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f8005040 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.725+0000 7fe702ffd700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7041067a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fe6f80089f0 tx=0x7fe6f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.725+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f8009320 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.725+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe6f8013ec0 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.725+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe704106e90 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.725+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6f8014d30 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.725+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe704107320 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.726+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe6f80095f0 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.726+0000 7fe6eb7fe700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe6ec031090 0x7fe6ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.726+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe704045bb0 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.726+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe6f80424a0 con 0x7fe7040f4850 2023-12-26T22:01:17.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.726+0000 7fe7027fc700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe6ec031090 0x7fe6ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:17.727 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.726+0000 7fe7027fc700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe6ec031090 0x7fe6ec033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fe6f40096a0 tx=0x7fe6f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:17.729 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.731+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe6f801d7f0 con 0x7fe7040f4850 2023-12-26T22:01:17.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.942+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7041081a0 con 0x7fe7040f4850 2023-12-26T22:01:17.942 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.943+0000 7fe6eb7fe700 1 -- 172.21.15.76:0/2302783847 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe6f801d7f0 con 0x7fe7040f4850 2023-12-26T22:01:17.942 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:17.942 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:17.943 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe6ec031090 msgr2=0x7fe6ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe6ec031090 0x7fe6ec033550 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fe6f40096a0 tx=0x7fe6f4006b40).stop 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 msgr2=0x7fe7041067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7041067a0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fe6f80089f0 tx=0x7fe6f800b040).stop 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 shutdown_connections 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe6ec031090 0x7fe6ec033550 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 --2- 172.21.15.76:0/2302783847 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7040f4850 0x7fe7041067a0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 >> 172.21.15.76:0/2302783847 conn(0x7fe7040f0390 msgr2=0x7fe7040f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 shutdown_connections 2023-12-26T22:01:17.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:17.944+0000 7fe7093a9700 1 -- 172.21.15.76:0/2302783847 wait complete. 2023-12-26T22:01:17.945 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:19.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:17.367198+0000 mgr.smithi059.gwkiyc (mgr.14156) 138 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:19.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:17.943321+0000 mon.smithi059 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.76:0/2302783847' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:19.329 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:19.330 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/498505417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 msgr2=0x7f2e380f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 --2- 172.21.15.76:0/498505417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e380f58d0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f2e20009fb0 tx=0x7f2e20009b30).stop 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/498505417 shutdown_connections 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 --2- 172.21.15.76:0/498505417 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e380f58d0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/498505417 >> 172.21.15.76:0/498505417 conn(0x7f2e380f0c10 msgr2=0x7f2e380f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/498505417 shutdown_connections 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.514+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/498505417 wait complete. 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3cb1e700 1 Processor -- start 2023-12-26T22:01:20.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3cb1e700 1 -- start start 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3cb1e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e381056a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3cb1e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e38105b90 con 0x7f2e380f54b0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e381056a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e381056a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:42770/0 (socket says 172.21.15.76:42770) 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3659c700 1 -- 172.21.15.76:0/3066191105 learned_addr learned my addr 172.21.15.76:0/3066191105 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3659c700 1 -- 172.21.15.76:0/3066191105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e20005040 con 0x7f2e380f54b0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.515+0000 7f2e3659c700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e381056a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f2e200089f0 tx=0x7f2e2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.516+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e20009320 con 0x7f2e380f54b0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.516+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2e20013ec0 con 0x7f2e380f54b0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.516+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e20014d30 con 0x7f2e380f54b0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.516+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e38105d90 con 0x7f2e380f54b0 2023-12-26T22:01:20.516 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.516+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e38106220 con 0x7f2e380f54b0 2023-12-26T22:01:20.517 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.516+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2e200095f0 con 0x7f2e380f54b0 2023-12-26T22:01:20.517 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.517+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e38045bb0 con 0x7f2e380f54b0 2023-12-26T22:01:20.517 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.517+0000 7f2e2effd700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2e24031090 0x7f2e24033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:20.517 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.517+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2e200424a0 con 0x7f2e380f54b0 2023-12-26T22:01:20.517 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.517+0000 7f2e35d9b700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2e24031090 0x7f2e24033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:20.517 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.517+0000 7f2e35d9b700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2e24031090 0x7f2e24033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2e280096a0 tx=0x7f2e28006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:20.520 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.522+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2e2001d7f0 con 0x7f2e380f54b0 2023-12-26T22:01:20.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.727+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e381070d0 con 0x7f2e380f54b0 2023-12-26T22:01:20.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.727+0000 7f2e2effd700 1 -- 172.21.15.76:0/3066191105 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2e2001d7f0 con 0x7f2e380f54b0 2023-12-26T22:01:20.727 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:20.727 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2e24031090 msgr2=0x7f2e24033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2e24031090 0x7f2e24033550 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f2e280096a0 tx=0x7f2e28006b40).stop 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 msgr2=0x7f2e381056a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e381056a0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f2e200089f0 tx=0x7f2e2000b040).stop 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 shutdown_connections 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2e24031090 0x7f2e24033550 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:20.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 --2- 172.21.15.76:0/3066191105 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2e380f54b0 0x7f2e381056a0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:20.729 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 >> 172.21.15.76:0/3066191105 conn(0x7f2e380f0c10 msgr2=0x7f2e380f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:20.729 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 shutdown_connections 2023-12-26T22:01:20.729 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:20.729+0000 7f2e3cb1e700 1 -- 172.21.15.76:0/3066191105 wait complete. 2023-12-26T22:01:20.729 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:19.367440+0000 mgr.smithi059.gwkiyc (mgr.14156) 139 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:20.727919+0000 mon.smithi059 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.76:0/3066191105' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:22.129 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:22.130 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:21.367768+0000 mgr.smithi059.gwkiyc (mgr.14156) 140 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.359+0000 7fa2f80e2700 1 -- 172.21.15.76:0/1162067462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 msgr2=0x7fa2f00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.359+0000 7fa2f80e2700 1 --2- 172.21.15.76:0/1162067462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f00f58a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fa2e0009fb0 tx=0x7fa2e0009b30).stop 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.359+0000 7fa2f80e2700 1 -- 172.21.15.76:0/1162067462 shutdown_connections 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.359+0000 7fa2f80e2700 1 --2- 172.21.15.76:0/1162067462 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f00f58a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.359+0000 7fa2f80e2700 1 -- 172.21.15.76:0/1162067462 >> 172.21.15.76:0/1162067462 conn(0x7fa2f00f0be0 msgr2=0x7fa2f00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f80e2700 1 -- 172.21.15.76:0/1162067462 shutdown_connections 2023-12-26T22:01:23.360 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f80e2700 1 -- 172.21.15.76:0/1162067462 wait complete. 2023-12-26T22:01:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f80e2700 1 Processor -- start 2023-12-26T22:01:23.361 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f80e2700 1 -- start start 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f80e2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f01056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f80e2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2f0105be0 con 0x7fa2f00f5480 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f5e7e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f01056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f5e7e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f01056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:42786/0 (socket says 172.21.15.76:42786) 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.360+0000 7fa2f5e7e700 1 -- 172.21.15.76:0/3656723783 learned_addr learned my addr 172.21.15.76:0/3656723783 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2f5e7e700 1 -- 172.21.15.76:0/3656723783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2e0005040 con 0x7fa2f00f5480 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2f5e7e700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f01056f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fa2e0000c00 tx=0x7fa2e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e00095f0 con 0x7fa2f00f5480 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2f0105de0 con 0x7fa2f00f5480 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa2e0007e50 con 0x7fa2f00f5480 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2e0014e70 con 0x7fa2f00f5480 2023-12-26T22:01:23.362 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.361+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2f0106270 con 0x7fa2f00f5480 2023-12-26T22:01:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.362+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa2e0027070 con 0x7fa2f00f5480 2023-12-26T22:01:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.362+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2f0045bb0 con 0x7fa2f00f5480 2023-12-26T22:01:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.362+0000 7fa2e67fc700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2dc0310e0 0x7fa2dc0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:23.363 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.362+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2e0043bc0 con 0x7fa2f00f5480 2023-12-26T22:01:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.362+0000 7fa2f567d700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2dc0310e0 0x7fa2dc0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:23.364 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.363+0000 7fa2f567d700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2dc0310e0 0x7fa2dc0335a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa2ec0096a0 tx=0x7fa2ec006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:23.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.367+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2e00079c0 con 0x7fa2f00f5480 2023-12-26T22:01:23.576 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.576+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa2f00faf70 con 0x7fa2f00f5480 2023-12-26T22:01:23.577 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.577+0000 7fa2e67fc700 1 -- 172.21.15.76:0/3656723783 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2e0027300 con 0x7fa2f00f5480 2023-12-26T22:01:23.577 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:23.577 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:23.578 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2dc0310e0 msgr2=0x7fa2dc0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:23.578 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2dc0310e0 0x7fa2dc0335a0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fa2ec0096a0 tx=0x7fa2ec006b40).stop 2023-12-26T22:01:23.578 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 msgr2=0x7fa2f01056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f01056f0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fa2e0000c00 tx=0x7fa2e000b040).stop 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 shutdown_connections 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2dc0310e0 0x7fa2dc0335a0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 --2- 172.21.15.76:0/3656723783 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa2f00f5480 0x7fa2f01056f0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 >> 172.21.15.76:0/3656723783 conn(0x7fa2f00f0be0 msgr2=0x7fa2f00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 shutdown_connections 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:23.578+0000 7fa2f80e2700 1 -- 172.21.15.76:0/3656723783 wait complete. 2023-12-26T22:01:23.579 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:24.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:23.578468+0000 mon.smithi059 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.76:0/3656723783' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:24.972 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:24.972 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:23.367994+0000 mgr.smithi059.gwkiyc (mgr.14156) 141 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 -- 172.21.15.76:0/3883258127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 msgr2=0x7fc0600f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 --2- 172.21.15.76:0/3883258127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0600f58a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fc05c009fb0 tx=0x7fc05c009b30).stop 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 -- 172.21.15.76:0/3883258127 shutdown_connections 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 --2- 172.21.15.76:0/3883258127 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0600f58a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 -- 172.21.15.76:0/3883258127 >> 172.21.15.76:0/3883258127 conn(0x7fc0600f0be0 msgr2=0x7fc0600f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 -- 172.21.15.76:0/3883258127 shutdown_connections 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 -- 172.21.15.76:0/3883258127 wait complete. 2023-12-26T22:01:26.105 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 Processor -- start 2023-12-26T22:01:26.106 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.104+0000 7fc067e33700 1 -- start start 2023-12-26T22:01:26.106 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc067e33700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0601089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:26.106 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc067e33700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc060108ed0 con 0x7fc0600f5480 2023-12-26T22:01:26.106 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc065bcf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0601089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:26.106 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc065bcf700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0601089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48482/0 (socket says 172.21.15.76:48482) 2023-12-26T22:01:26.106 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc065bcf700 1 -- 172.21.15.76:0/1868616349 learned_addr learned my addr 172.21.15.76:0/1868616349 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:26.107 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc065bcf700 1 -- 172.21.15.76:0/1868616349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc05c005040 con 0x7fc0600f5480 2023-12-26T22:01:26.107 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.105+0000 7fc065bcf700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0601089e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fc05c0089f0 tx=0x7fc05c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:26.107 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc05c009320 con 0x7fc0600f5480 2023-12-26T22:01:26.107 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0601090d0 con 0x7fc0600f5480 2023-12-26T22:01:26.107 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc05c013ec0 con 0x7fc0600f5480 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc05c014d30 con 0x7fc0600f5480 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc060109560 con 0x7fc0600f5480 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc05c0095f0 con 0x7fc0600f5480 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.106+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc060045bb0 con 0x7fc0600f5480 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.107+0000 7fc0567fc700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc04c0310e0 0x7fc04c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.107+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc05c042450 con 0x7fc0600f5480 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.107+0000 7fc0653ce700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc04c0310e0 0x7fc04c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:26.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.107+0000 7fc0653ce700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc04c0310e0 0x7fc04c0335a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:26.111 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.111+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc05c01d980 con 0x7fc0600f5480 2023-12-26T22:01:26.317 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.316+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc06010a410 con 0x7fc0600f5480 2023-12-26T22:01:26.317 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.317+0000 7fc0567fc700 1 -- 172.21.15.76:0/1868616349 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc05c01d380 con 0x7fc0600f5480 2023-12-26T22:01:26.317 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:26.318 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc04c0310e0 msgr2=0x7fc04c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc04c0310e0 0x7fc04c0335a0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fc0500096a0 tx=0x7fc050006b40).stop 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 msgr2=0x7fc0601089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0601089e0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7fc05c0089f0 tx=0x7fc05c00b040).stop 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 shutdown_connections 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc04c0310e0 0x7fc04c0335a0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:26.319 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 --2- 172.21.15.76:0/1868616349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc0600f5480 0x7fc0601089e0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 >> 172.21.15.76:0/1868616349 conn(0x7fc0600f0be0 msgr2=0x7fc06005ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 shutdown_connections 2023-12-26T22:01:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:26.319+0000 7fc067e33700 1 -- 172.21.15.76:0/1868616349 wait complete. 2023-12-26T22:01:26.320 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:27.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:25.368276+0000 mgr.smithi059.gwkiyc (mgr.14156) 142 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:27.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:26.318733+0000 mon.smithi059 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.76:0/1868616349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:27.727 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:27.728 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:28.975 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 -- 172.21.15.76:0/2502616155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 msgr2=0x7fcd380f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:28.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 --2- 172.21.15.76:0/2502616155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd380f58a0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7fcd28009fb0 tx=0x7fcd28009b30).stop 2023-12-26T22:01:28.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 -- 172.21.15.76:0/2502616155 shutdown_connections 2023-12-26T22:01:28.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 --2- 172.21.15.76:0/2502616155 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd380f58a0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:28.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 -- 172.21.15.76:0/2502616155 >> 172.21.15.76:0/2502616155 conn(0x7fcd380f0be0 msgr2=0x7fcd380f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:28.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 -- 172.21.15.76:0/2502616155 shutdown_connections 2023-12-26T22:01:28.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.975+0000 7fcd3e39f700 1 -- 172.21.15.76:0/2502616155 wait complete. 2023-12-26T22:01:28.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.976+0000 7fcd3e39f700 1 Processor -- start 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.976+0000 7fcd3e39f700 1 -- start start 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.976+0000 7fcd3e39f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd381056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.976+0000 7fcd3e39f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd38105be0 con 0x7fcd380f5480 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd37fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd381056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd37fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd381056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48486/0 (socket says 172.21.15.76:48486) 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd37fff700 1 -- 172.21.15.76:0/3903291348 learned_addr learned my addr 172.21.15.76:0/3903291348 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd37fff700 1 -- 172.21.15.76:0/3903291348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd28005040 con 0x7fcd380f5480 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd37fff700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd381056f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fcd280089f0 tx=0x7fcd2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd28009320 con 0x7fcd380f5480 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd38105de0 con 0x7fcd380f5480 2023-12-26T22:01:28.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fcd28013ec0 con 0x7fcd380f5480 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd28014d30 con 0x7fcd380f5480 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.977+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd38106270 con 0x7fcd380f5480 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.978+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fcd280263a0 con 0x7fcd380f5480 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.978+0000 7fcd34ff9700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcd20031090 0x7fcd20033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.978+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fcd280423a0 con 0x7fcd380f5480 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.978+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd38045bb0 con 0x7fcd380f5480 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.978+0000 7fcd377fe700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcd20031090 0x7fcd20033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:28.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.979+0000 7fcd377fe700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcd20031090 0x7fcd20033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fcd2c0096a0 tx=0x7fcd2c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:28.983 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:28.983+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fcd280266d0 con 0x7fcd380f5480 2023-12-26T22:01:29.191 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.191+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd380faf70 con 0x7fcd380f5480 2023-12-26T22:01:29.191 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.191+0000 7fcd34ff9700 1 -- 172.21.15.76:0/3903291348 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fcd2802b020 con 0x7fcd380f5480 2023-12-26T22:01:29.192 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:29.192 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:29.193 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcd20031090 msgr2=0x7fcd20033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:29.193 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcd20031090 0x7fcd20033550 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fcd2c0096a0 tx=0x7fcd2c006b40).stop 2023-12-26T22:01:29.193 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 msgr2=0x7fcd381056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:29.193 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd381056f0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7fcd280089f0 tx=0x7fcd2800b040).stop 2023-12-26T22:01:29.193 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 shutdown_connections 2023-12-26T22:01:29.194 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fcd20031090 0x7fcd20033550 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:29.194 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 --2- 172.21.15.76:0/3903291348 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fcd380f5480 0x7fcd381056f0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:29.194 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 >> 172.21.15.76:0/3903291348 conn(0x7fcd380f0be0 msgr2=0x7fcd380f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:29.194 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 shutdown_connections 2023-12-26T22:01:29.194 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:29.193+0000 7fcd3e39f700 1 -- 172.21.15.76:0/3903291348 wait complete. 2023-12-26T22:01:29.194 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:27.368564+0000 mgr.smithi059.gwkiyc (mgr.14156) 143 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:30.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:29 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:29.193088+0000 mon.smithi059 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.76:0/3903291348' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:30.597 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:30.597 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:29.368922+0000 mgr.smithi059.gwkiyc (mgr.14156) 144 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:31.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.798+0000 7fbe18211700 1 -- 172.21.15.76:0/2570498754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 msgr2=0x7fbe100f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:31.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.798+0000 7fbe18211700 1 --2- 172.21.15.76:0/2570498754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe100f58a0 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fbe00009fb0 tx=0x7fbe00009b30).stop 2023-12-26T22:01:31.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.798+0000 7fbe18211700 1 -- 172.21.15.76:0/2570498754 shutdown_connections 2023-12-26T22:01:31.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.798+0000 7fbe18211700 1 --2- 172.21.15.76:0/2570498754 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe100f58a0 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:31.800 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.798+0000 7fbe18211700 1 -- 172.21.15.76:0/2570498754 >> 172.21.15.76:0/2570498754 conn(0x7fbe100f0be0 msgr2=0x7fbe100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe18211700 1 -- 172.21.15.76:0/2570498754 shutdown_connections 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe18211700 1 -- 172.21.15.76:0/2570498754 wait complete. 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe18211700 1 Processor -- start 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe18211700 1 -- start start 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe18211700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe101067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe18211700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe10106cd0 con 0x7fbe100f5480 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe15fad700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe101067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe15fad700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe101067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48502/0 (socket says 172.21.15.76:48502) 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.799+0000 7fbe15fad700 1 -- 172.21.15.76:0/1898421984 learned_addr learned my addr 172.21.15.76:0/1898421984 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe15fad700 1 -- 172.21.15.76:0/1898421984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe00005040 con 0x7fbe100f5480 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe15fad700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe101067e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fbe000089f0 tx=0x7fbe0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe00009320 con 0x7fbe100f5480 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe10106ed0 con 0x7fbe100f5480 2023-12-26T22:01:31.801 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbe00013ec0 con 0x7fbe100f5480 2023-12-26T22:01:31.802 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe00014d30 con 0x7fbe100f5480 2023-12-26T22:01:31.802 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.800+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe10107360 con 0x7fbe100f5480 2023-12-26T22:01:31.802 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.801+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbe00014420 con 0x7fbe100f5480 2023-12-26T22:01:31.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.801+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe10045bb0 con 0x7fbe100f5480 2023-12-26T22:01:31.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.801+0000 7fbe067fc700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbdfc031090 0x7fbdfc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:31.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.801+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbe000424c0 con 0x7fbe100f5480 2023-12-26T22:01:31.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.801+0000 7fbe157ac700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbdfc031090 0x7fbdfc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:31.803 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.802+0000 7fbe157ac700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbdfc031090 0x7fbdfc033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fbe0c0096a0 tx=0x7fbe0c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:31.806 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:31.806+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbe0001d980 con 0x7fbe100f5480 2023-12-26T22:01:32.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.012+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe10108210 con 0x7fbe100f5480 2023-12-26T22:01:32.014 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.013+0000 7fbe067fc700 1 -- 172.21.15.76:0/1898421984 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbe0001d380 con 0x7fbe100f5480 2023-12-26T22:01:32.014 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:32.014 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:32.014 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbdfc031090 msgr2=0x7fbdfc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:32.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbdfc031090 0x7fbdfc033550 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fbe0c0096a0 tx=0x7fbe0c006b40).stop 2023-12-26T22:01:32.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 msgr2=0x7fbe101067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:32.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe101067e0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fbe000089f0 tx=0x7fbe0000b040).stop 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 shutdown_connections 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbdfc031090 0x7fbdfc033550 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 --2- 172.21.15.76:0/1898421984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbe100f5480 0x7fbe101067e0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 >> 172.21.15.76:0/1898421984 conn(0x7fbe100f0be0 msgr2=0x7fbe1005ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 shutdown_connections 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:32.015+0000 7fbe18211700 1 -- 172.21.15.76:0/1898421984 wait complete. 2023-12-26T22:01:32.016 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:31.369322+0000 mgr.smithi059.gwkiyc (mgr.14156) 145 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:32.014798+0000 mon.smithi059 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.76:0/1898421984' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:33.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:33.397 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 -- 172.21.15.76:0/2259729998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 msgr2=0x7fc8840f9200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 --2- 172.21.15.76:0/2259729998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f9200 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fc878009fb0 tx=0x7fc878009b30).stop 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 -- 172.21.15.76:0/2259729998 shutdown_connections 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 --2- 172.21.15.76:0/2259729998 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f9200 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 -- 172.21.15.76:0/2259729998 >> 172.21.15.76:0/2259729998 conn(0x7fc8840f0be0 msgr2=0x7fc8840f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 -- 172.21.15.76:0/2259729998 shutdown_connections 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.569+0000 7fc88ab4f700 1 -- 172.21.15.76:0/2259729998 wait complete. 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc88ab4f700 1 Processor -- start 2023-12-26T22:01:34.570 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc88ab4f700 1 -- start start 2023-12-26T22:01:34.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc88ab4f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f6c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:34.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc88ab4f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8840f5100 con 0x7fc8840602c0 2023-12-26T22:01:34.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc8888eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f6c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:34.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc8888eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f6c80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48526/0 (socket says 172.21.15.76:48526) 2023-12-26T22:01:34.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.570+0000 7fc8888eb700 1 -- 172.21.15.76:0/1934696473 learned_addr learned my addr 172.21.15.76:0/1934696473 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:34.572 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc8888eb700 1 -- 172.21.15.76:0/1934696473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc878005040 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc8888eb700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f6c80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fc878013fd0 tx=0x7fc878009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc878018070 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8840f52a0 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc878007b10 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc87801c3c0 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.571+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8840f5730 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.572+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc87801c640 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.572+0000 7fc8817fa700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc874031090 0x7fc874033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.572+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc878042a90 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.572+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8840591e0 con 0x7fc8840602c0 2023-12-26T22:01:34.573 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.572+0000 7fc883fff700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc874031090 0x7fc874033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:34.574 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.573+0000 7fc883fff700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc874031090 0x7fc874033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc8700096a0 tx=0x7fc870006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:34.577 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.577+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc878024300 con 0x7fc8840602c0 2023-12-26T22:01:34.786 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.786+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc8840fc070 con 0x7fc8840602c0 2023-12-26T22:01:34.787 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.787+0000 7fc8817fa700 1 -- 172.21.15.76:0/1934696473 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc87801bb60 con 0x7fc8840602c0 2023-12-26T22:01:34.787 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:34.787 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.788+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc874031090 msgr2=0x7fc874033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.788+0000 7fc88ab4f700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc874031090 0x7fc874033550 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fc8700096a0 tx=0x7fc870006b40).stop 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.788+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 msgr2=0x7fc8840f6c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.788+0000 7fc88ab4f700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f6c80 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fc878013fd0 tx=0x7fc878009b30).stop 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.789+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 shutdown_connections 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.789+0000 7fc88ab4f700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc874031090 0x7fc874033550 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:34.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.789+0000 7fc88ab4f700 1 --2- 172.21.15.76:0/1934696473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc8840602c0 0x7fc8840f6c80 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.789+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 >> 172.21.15.76:0/1934696473 conn(0x7fc8840f0be0 msgr2=0x7fc8840f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.789+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 shutdown_connections 2023-12-26T22:01:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:34.789+0000 7fc88ab4f700 1 -- 172.21.15.76:0/1934696473 wait complete. 2023-12-26T22:01:34.790 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:33.369684+0000 mgr.smithi059.gwkiyc (mgr.14156) 146 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:34.788369+0000 mon.smithi059 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.76:0/1934696473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:36.174 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:36.174 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:35.370052+0000 mgr.smithi059.gwkiyc (mgr.14156) 147 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 -- 172.21.15.76:0/4279473774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 msgr2=0x7f6ccc0f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 --2- 172.21.15.76:0/4279473774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc0f9160 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f6cbc009fb0 tx=0x7f6cbc009b30).stop 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 -- 172.21.15.76:0/4279473774 shutdown_connections 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 --2- 172.21.15.76:0/4279473774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc0f9160 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 -- 172.21.15.76:0/4279473774 >> 172.21.15.76:0/4279473774 conn(0x7f6ccc0f0be0 msgr2=0x7f6ccc0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 -- 172.21.15.76:0/4279473774 shutdown_connections 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.338+0000 7f6cd4172700 1 -- 172.21.15.76:0/4279473774 wait complete. 2023-12-26T22:01:37.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd4172700 1 Processor -- start 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd4172700 1 -- start start 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd4172700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd4172700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ccc106c40 con 0x7f6ccc060360 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd1f0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd1f0e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44318/0 (socket says 172.21.15.76:44318) 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd1f0e700 1 -- 172.21.15.76:0/3305466007 learned_addr learned my addr 172.21.15.76:0/3305466007 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd1f0e700 1 -- 172.21.15.76:0/3305466007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc005040 con 0x7f6ccc060360 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.339+0000 7f6cd1f0e700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc106750 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6cbc000c00 tx=0x7f6cbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.340+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cbc0095f0 con 0x7f6ccc060360 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.340+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ccc106e40 con 0x7f6ccc060360 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.340+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6cbc007e50 con 0x7f6ccc060360 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.340+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6cbc014e70 con 0x7f6ccc060360 2023-12-26T22:01:37.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.340+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ccc1072d0 con 0x7f6ccc060360 2023-12-26T22:01:37.345 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.341+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6cbc025070 con 0x7f6ccc060360 2023-12-26T22:01:37.345 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.341+0000 7f6cc27fc700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6cb8031090 0x7f6cb8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:37.345 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.341+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ccc045bb0 con 0x7f6ccc060360 2023-12-26T22:01:37.345 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.341+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6cbc041d70 con 0x7f6ccc060360 2023-12-26T22:01:37.345 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.341+0000 7f6cd170d700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6cb8031090 0x7f6cb8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:37.345 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.341+0000 7f6cd170d700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6cb8031090 0x7f6cb8033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:37.346 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.346+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6cbc00f4e0 con 0x7f6ccc060360 2023-12-26T22:01:37.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.558+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6ccc108150 con 0x7f6ccc060360 2023-12-26T22:01:37.559 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.559+0000 7f6cc27fc700 1 -- 172.21.15.76:0/3305466007 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6cbc00f300 con 0x7f6ccc060360 2023-12-26T22:01:37.559 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:37.559 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:37.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6cb8031090 msgr2=0x7f6cb8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6cb8031090 0x7f6cb8033550 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f6cc80096a0 tx=0x7f6cc8006b40).stop 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 msgr2=0x7f6ccc106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc106750 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f6cbc000c00 tx=0x7f6cbc00b040).stop 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 shutdown_connections 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6cb8031090 0x7f6cb8033550 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 --2- 172.21.15.76:0/3305466007 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f6ccc060360 0x7f6ccc106750 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 >> 172.21.15.76:0/3305466007 conn(0x7f6ccc0f0be0 msgr2=0x7f6ccc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 shutdown_connections 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:37.560+0000 7f6cd4172700 1 -- 172.21.15.76:0/3305466007 wait complete. 2023-12-26T22:01:37.561 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:38.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:37 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:37.560076+0000 mon.smithi059 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.76:0/3305466007' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:38.893 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:38.893 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:39.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:37.370364+0000 mgr.smithi059.gwkiyc (mgr.14156) 148 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.040+0000 7f23aee30700 1 -- 172.21.15.76:0/1393450863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 msgr2=0x7f23a80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.040+0000 7f23aee30700 1 --2- 172.21.15.76:0/1393450863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a80f58a0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f239c009fb0 tx=0x7f239c009b30).stop 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.040+0000 7f23aee30700 1 -- 172.21.15.76:0/1393450863 shutdown_connections 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.040+0000 7f23aee30700 1 --2- 172.21.15.76:0/1393450863 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a80f58a0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.040+0000 7f23aee30700 1 -- 172.21.15.76:0/1393450863 >> 172.21.15.76:0/1393450863 conn(0x7f23a80f0be0 msgr2=0x7f23a80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.041+0000 7f23aee30700 1 -- 172.21.15.76:0/1393450863 shutdown_connections 2023-12-26T22:01:40.040 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.041+0000 7f23aee30700 1 -- 172.21.15.76:0/1393450863 wait complete. 2023-12-26T22:01:40.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.041+0000 7f23aee30700 1 Processor -- start 2023-12-26T22:01:40.041 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.041+0000 7f23aee30700 1 -- start start 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23aee30700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a81067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23aee30700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23a8106cd0 con 0x7f23a80f5480 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23acbcc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a81067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23acbcc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a81067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44342/0 (socket says 172.21.15.76:44342) 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23acbcc700 1 -- 172.21.15.76:0/4221914786 learned_addr learned my addr 172.21.15.76:0/4221914786 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23acbcc700 1 -- 172.21.15.76:0/4221914786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f239c005040 con 0x7f23a80f5480 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.042+0000 7f23acbcc700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a81067e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f239c0089f0 tx=0x7f239c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.043+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c009320 con 0x7f23a80f5480 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.043+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23a8106ed0 con 0x7f23a80f5480 2023-12-26T22:01:40.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.043+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f239c013ec0 con 0x7f23a80f5480 2023-12-26T22:01:40.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.043+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f239c014d30 con 0x7f23a80f5480 2023-12-26T22:01:40.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.043+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23a8107360 con 0x7f23a80f5480 2023-12-26T22:01:40.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.043+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f239c014420 con 0x7f23a80f5480 2023-12-26T22:01:40.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.044+0000 7f23a57fa700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2398031090 0x7f2398033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.044+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23a8045bb0 con 0x7f23a80f5480 2023-12-26T22:01:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.044+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f239c0424a0 con 0x7f23a80f5480 2023-12-26T22:01:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.044+0000 7f23a7fff700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2398031090 0x7f2398033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:40.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.044+0000 7f23a7fff700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2398031090 0x7f2398033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f23940096a0 tx=0x7f2394006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:40.048 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.049+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f239c014750 con 0x7f23a80f5480 2023-12-26T22:01:40.256 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.257+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23a80faf70 con 0x7f23a80f5480 2023-12-26T22:01:40.257 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.258+0000 7f23a57fa700 1 -- 172.21.15.76:0/4221914786 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f239c01d7f0 con 0x7f23a80f5480 2023-12-26T22:01:40.257 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:40.257 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.259+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2398031090 msgr2=0x7f2398033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.259+0000 7f23aee30700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2398031090 0x7f2398033550 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f23940096a0 tx=0x7f2394006b40).stop 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.259+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 msgr2=0x7f23a81067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.259+0000 7f23aee30700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a81067e0 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f239c0089f0 tx=0x7f239c00b040).stop 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.260+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 shutdown_connections 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.260+0000 7f23aee30700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2398031090 0x7f2398033550 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.260+0000 7f23aee30700 1 --2- 172.21.15.76:0/4221914786 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f23a80f5480 0x7f23a81067e0 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.260+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 >> 172.21.15.76:0/4221914786 conn(0x7f23a80f0be0 msgr2=0x7f23a805ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.260+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 shutdown_connections 2023-12-26T22:01:40.259 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:40.260+0000 7f23aee30700 1 -- 172.21.15.76:0/4221914786 wait complete. 2023-12-26T22:01:40.260 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:39.370638+0000 mgr.smithi059.gwkiyc (mgr.14156) 149 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:40.258676+0000 mon.smithi059 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.76:0/4221914786' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:41.665 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:41.666 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:42.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 -- 172.21.15.76:0/1018252411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 msgr2=0x7f57bc0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:42.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 --2- 172.21.15.76:0/1018252411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc0f69b0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f57a4009fb0 tx=0x7f57a4009b30).stop 2023-12-26T22:01:42.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 -- 172.21.15.76:0/1018252411 shutdown_connections 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 --2- 172.21.15.76:0/1018252411 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc0f69b0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 -- 172.21.15.76:0/1018252411 >> 172.21.15.76:0/1018252411 conn(0x7f57bc0f1cf0 msgr2=0x7f57bc0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 -- 172.21.15.76:0/1018252411 shutdown_connections 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.906+0000 7f57c1711700 1 -- 172.21.15.76:0/1018252411 wait complete. 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57c1711700 1 Processor -- start 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57c1711700 1 -- start start 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57c1711700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc106800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57c1711700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57bc106cf0 con 0x7f57bc0f6590 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57baffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc106800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:42.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57baffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc106800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44368/0 (socket says 172.21.15.76:44368) 2023-12-26T22:01:42.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57baffd700 1 -- 172.21.15.76:0/2768270699 learned_addr learned my addr 172.21.15.76:0/2768270699 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:42.909 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.907+0000 7f57baffd700 1 -- 172.21.15.76:0/2768270699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57a4005040 con 0x7f57bc0f6590 2023-12-26T22:01:42.909 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.908+0000 7f57baffd700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc106800 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f57a40089f0 tx=0x7f57a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:42.909 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.908+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a4009320 con 0x7f57bc0f6590 2023-12-26T22:01:42.909 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.908+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57bc106ef0 con 0x7f57bc0f6590 2023-12-26T22:01:42.909 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.908+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57bc107380 con 0x7f57bc0f6590 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.908+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f57a4013ec0 con 0x7f57bc0f6590 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.908+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57a4014d00 con 0x7f57bc0f6590 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.909+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f57a4014420 con 0x7f57bc0f6590 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.909+0000 7f57b37fe700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f57a8031090 0x7f57a8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.909+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57a0004fa0 con 0x7f57bc0f6590 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.909+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f57a4042390 con 0x7f57bc0f6590 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.909+0000 7f57ba7fc700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f57a8031090 0x7f57a8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:42.910 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.910+0000 7f57ba7fc700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f57a8031090 0x7f57a8033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f57ac0096a0 tx=0x7f57ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:42.913 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:42.914+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f57a4012d00 con 0x7f57bc0f6590 2023-12-26T22:01:43.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.119+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57a0005670 con 0x7f57bc0f6590 2023-12-26T22:01:43.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.119+0000 7f57b37fe700 1 -- 172.21.15.76:0/2768270699 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f57a4012b20 con 0x7f57bc0f6590 2023-12-26T22:01:43.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:43.118 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f57a8031090 msgr2=0x7f57a8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f57a8031090 0x7f57a8033550 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f57ac0096a0 tx=0x7f57ac006b40).stop 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 msgr2=0x7f57bc106800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc106800 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f57a40089f0 tx=0x7f57a400b040).stop 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 shutdown_connections 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f57a8031090 0x7f57a8033550 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 --2- 172.21.15.76:0/2768270699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57bc0f6590 0x7f57bc106800 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 >> 172.21.15.76:0/2768270699 conn(0x7f57bc0f1cf0 msgr2=0x7f57bc0f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:43.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 shutdown_connections 2023-12-26T22:01:43.121 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:43.121+0000 7f57c1711700 1 -- 172.21.15.76:0/2768270699 wait complete. 2023-12-26T22:01:43.121 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:41.370999+0000 mgr.smithi059.gwkiyc (mgr.14156) 150 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:44.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:43 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:43.119912+0000 mon.smithi059 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.76:0/2768270699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:44.500 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:44.500 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:45.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:43.371315+0000 mgr.smithi059.gwkiyc (mgr.14156) 151 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:45.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.718+0000 7fadb329a700 1 -- 172.21.15.76:0/199362713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 msgr2=0x7fadac0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:45.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.718+0000 7fadb329a700 1 --2- 172.21.15.76:0/199362713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac0f69b0 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7fad9c004760 tx=0x7fad9c009ad0).stop 2023-12-26T22:01:45.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 -- 172.21.15.76:0/199362713 shutdown_connections 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 --2- 172.21.15.76:0/199362713 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac0f69b0 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 -- 172.21.15.76:0/199362713 >> 172.21.15.76:0/199362713 conn(0x7fadac0f1cd0 msgr2=0x7fadac0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 -- 172.21.15.76:0/199362713 shutdown_connections 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 -- 172.21.15.76:0/199362713 wait complete. 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 Processor -- start 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 -- start start 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:45.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.719+0000 7fadb329a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fadac069f00 con 0x7fadac0f6590 2023-12-26T22:01:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fadb1036700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fadb1036700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48456/0 (socket says 172.21.15.76:48456) 2023-12-26T22:01:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fadb1036700 1 -- 172.21.15.76:0/2983421374 learned_addr learned my addr 172.21.15.76:0/2983421374 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fadb1036700 1 -- 172.21.15.76:0/2983421374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad9c009d70 con 0x7fadac0f6590 2023-12-26T22:01:45.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fadb1036700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac06ba80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fad9c013f10 tx=0x7fad9c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad9c00e040 con 0x7fadac0f6590 2023-12-26T22:01:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fadac06a0a0 con 0x7fadac0f6590 2023-12-26T22:01:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fad9c007df0 con 0x7fadac0f6590 2023-12-26T22:01:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.720+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fad9c014d60 con 0x7fadac0f6590 2023-12-26T22:01:45.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.721+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fadac06a530 con 0x7fadac0f6590 2023-12-26T22:01:45.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.722+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fad9c014520 con 0x7fadac0f6590 2023-12-26T22:01:45.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.722+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fadac0591e0 con 0x7fadac0f6590 2023-12-26T22:01:45.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.722+0000 7fada1ffb700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad98031090 0x7fad98033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:45.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.722+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fad9c042580 con 0x7fadac0f6590 2023-12-26T22:01:45.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.722+0000 7fadb0835700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad98031090 0x7fad98033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:45.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.723+0000 7fadb0835700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad98031090 0x7fad98033550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fada80096a0 tx=0x7fada8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:45.726 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.727+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fad9c007960 con 0x7fadac0f6590 2023-12-26T22:01:45.932 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.933+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fadac045bb0 con 0x7fadac0f6590 2023-12-26T22:01:45.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.934+0000 7fada1ffb700 1 -- 172.21.15.76:0/2983421374 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fad9c0262f0 con 0x7fadac0f6590 2023-12-26T22:01:45.933 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:45.933 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad98031090 msgr2=0x7fad98033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad98031090 0x7fad98033550 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fada80096a0 tx=0x7fada8006b40).stop 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 msgr2=0x7fadac06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac06ba80 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fad9c013f10 tx=0x7fad9c009ad0).stop 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 shutdown_connections 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fad98031090 0x7fad98033550 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:45.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 --2- 172.21.15.76:0/2983421374 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fadac0f6590 0x7fadac06ba80 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 >> 172.21.15.76:0/2983421374 conn(0x7fadac0f1cd0 msgr2=0x7fadac0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 shutdown_connections 2023-12-26T22:01:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:45.936+0000 7fadb329a700 1 -- 172.21.15.76:0/2983421374 wait complete. 2023-12-26T22:01:45.936 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:45.469625+0000 mon.smithi059 (mon.0) 213 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T22:01:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:45.469860+0000 mon.smithi059 (mon.0) 214 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T22:01:47.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:45.371589+0000 mgr.smithi059.gwkiyc (mgr.14156) 152 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:47.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:45.934570+0000 mon.smithi059 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.76:0/2983421374' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:47.359 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:47.360 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:48.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.628+0000 7fa15fa28700 1 -- 172.21.15.76:0/1330864008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 msgr2=0x7fa1580f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:48.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.628+0000 7fa15fa28700 1 --2- 172.21.15.76:0/1330864008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1580f69b0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fa14c004760 tx=0x7fa14c009ad0).stop 2023-12-26T22:01:48.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.628+0000 7fa15fa28700 1 -- 172.21.15.76:0/1330864008 shutdown_connections 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.628+0000 7fa15fa28700 1 --2- 172.21.15.76:0/1330864008 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1580f69b0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.628+0000 7fa15fa28700 1 -- 172.21.15.76:0/1330864008 >> 172.21.15.76:0/1330864008 conn(0x7fa1580f1cd0 msgr2=0x7fa1580f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15fa28700 1 -- 172.21.15.76:0/1330864008 shutdown_connections 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15fa28700 1 -- 172.21.15.76:0/1330864008 wait complete. 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15fa28700 1 Processor -- start 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15fa28700 1 -- start start 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15fa28700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1581067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15fa28700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa158106cb0 con 0x7fa1580f6590 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15d7c4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1581067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.629+0000 7fa15d7c4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1581067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48466/0 (socket says 172.21.15.76:48466) 2023-12-26T22:01:48.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa15d7c4700 1 -- 172.21.15.76:0/3405425253 learned_addr learned my addr 172.21.15.76:0/3405425253 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:48.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa15d7c4700 1 -- 172.21.15.76:0/3405425253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa14c009d70 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa15d7c4700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1581067c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fa14c013f10 tx=0x7fa14c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa14c00e040 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa14c007df0 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa158106eb0 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa14c014d60 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.630+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa158107340 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.631+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa14c01f460 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.631+0000 7fa149ffb700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa144031090 0x7fa144033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.631+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa14c042540 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.632+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1580591e0 con 0x7fa1580f6590 2023-12-26T22:01:48.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.632+0000 7fa15cfc3700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa144031090 0x7fa144033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:48.632 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.632+0000 7fa15cfc3700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa144031090 0x7fa144033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa1540096a0 tx=0x7fa154006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:48.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.636+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa14c0283f0 con 0x7fa1580f6590 2023-12-26T22:01:48.841 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.842+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa158045bb0 con 0x7fa1580f6590 2023-12-26T22:01:48.842 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.843+0000 7fa149ffb700 1 -- 172.21.15.76:0/3405425253 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa14c012330 con 0x7fa1580f6590 2023-12-26T22:01:48.842 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:48.842 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa144031090 msgr2=0x7fa144033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa144031090 0x7fa144033550 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fa1540096a0 tx=0x7fa154006b40).stop 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 msgr2=0x7fa1581067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1581067c0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7fa14c013f10 tx=0x7fa14c009ad0).stop 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 shutdown_connections 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa144031090 0x7fa144033550 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 --2- 172.21.15.76:0/3405425253 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa1580f6590 0x7fa1581067c0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 >> 172.21.15.76:0/3405425253 conn(0x7fa1580f1cd0 msgr2=0x7fa1580f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 shutdown_connections 2023-12-26T22:01:48.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:48.845+0000 7fa15fa28700 1 -- 172.21.15.76:0/3405425253 wait complete. 2023-12-26T22:01:48.845 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:47.371926+0000 mgr.smithi059.gwkiyc (mgr.14156) 153 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:48.843699+0000 mon.smithi059 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.76:0/3405425253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:50.225 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:50.225 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:49.372313+0000 mgr.smithi059.gwkiyc (mgr.14156) 154 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 -- 172.21.15.76:0/3154201555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 msgr2=0x7f53140f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 --2- 172.21.15.76:0/3154201555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f53140f56f0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f5310009fb0 tx=0x7f5310009b30).stop 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 -- 172.21.15.76:0/3154201555 shutdown_connections 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 --2- 172.21.15.76:0/3154201555 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f53140f56f0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 -- 172.21.15.76:0/3154201555 >> 172.21.15.76:0/3154201555 conn(0x7f53140f0a30 msgr2=0x7f53140f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 -- 172.21.15.76:0/3154201555 shutdown_connections 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.402+0000 7f531a203700 1 -- 172.21.15.76:0/3154201555 wait complete. 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.403+0000 7f531a203700 1 Processor -- start 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.403+0000 7f531a203700 1 -- start start 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.403+0000 7f531a203700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f5314105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.403+0000 7f531a203700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5314105a00 con 0x7f53140f52d0 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.403+0000 7f5319201700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f5314105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:51.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5319201700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f5314105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48488/0 (socket says 172.21.15.76:48488) 2023-12-26T22:01:51.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5319201700 1 -- 172.21.15.76:0/4224173042 learned_addr learned my addr 172.21.15.76:0/4224173042 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:51.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5319201700 1 -- 172.21.15.76:0/4224173042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5310005040 con 0x7f53140f52d0 2023-12-26T22:01:51.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5319201700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f5314105510 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f5310000c00 tx=0x7f531000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53100095f0 con 0x7f53140f52d0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5310007e50 con 0x7f53140f52d0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5314105c00 con 0x7f53140f52d0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5310014e70 con 0x7f53140f52d0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.404+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5314106090 con 0x7f53140f52d0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.405+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5310025070 con 0x7f53140f52d0 2023-12-26T22:01:51.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.405+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5314045bb0 con 0x7f53140f52d0 2023-12-26T22:01:51.406 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.405+0000 7f5309ffb700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5300031090 0x7f5300033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:51.406 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.406+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5310041d10 con 0x7f53140f52d0 2023-12-26T22:01:51.406 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.406+0000 7f5318a00700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5300031090 0x7f5300033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:51.406 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.406+0000 7f5318a00700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5300031090 0x7f5300033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f53040096a0 tx=0x7f5304006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:51.409 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.410+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f531000f4e0 con 0x7f53140f52d0 2023-12-26T22:01:51.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.619+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5314106e90 con 0x7f53140f52d0 2023-12-26T22:01:51.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.620+0000 7f5309ffb700 1 -- 172.21.15.76:0/4224173042 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f531000f300 con 0x7f53140f52d0 2023-12-26T22:01:51.619 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:51.619 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:51.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5300031090 msgr2=0x7f5300033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5300031090 0x7f5300033550 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f53040096a0 tx=0x7f5304006b40).stop 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 msgr2=0x7f5314105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f5314105510 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f5310000c00 tx=0x7f531000b040).stop 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 shutdown_connections 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5300031090 0x7f5300033550 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 --2- 172.21.15.76:0/4224173042 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f53140f52d0 0x7f5314105510 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 >> 172.21.15.76:0/4224173042 conn(0x7f53140f0a30 msgr2=0x7f53140f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.621+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 shutdown_connections 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:51.622+0000 7f531a203700 1 -- 172.21.15.76:0/4224173042 wait complete. 2023-12-26T22:01:51.621 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:52.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:51 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:51.620290+0000 mon.smithi059 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.76:0/4224173042' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:52.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:52.991 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:51.372594+0000 mgr.smithi059.gwkiyc (mgr.14156) 155 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.169+0000 7f67068d5700 1 -- 172.21.15.76:0/50427447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 msgr2=0x7f67000f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.169+0000 7f67068d5700 1 --2- 172.21.15.76:0/50427447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67000f58a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f66f0009fb0 tx=0x7f66f0009b30).stop 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.169+0000 7f67068d5700 1 -- 172.21.15.76:0/50427447 shutdown_connections 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.169+0000 7f67068d5700 1 --2- 172.21.15.76:0/50427447 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67000f58a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.169+0000 7f67068d5700 1 -- 172.21.15.76:0/50427447 >> 172.21.15.76:0/50427447 conn(0x7f67000f0be0 msgr2=0x7f67000f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.170+0000 7f67068d5700 1 -- 172.21.15.76:0/50427447 shutdown_connections 2023-12-26T22:01:54.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.170+0000 7f67068d5700 1 -- 172.21.15.76:0/50427447 wait complete. 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.170+0000 7f67068d5700 1 Processor -- start 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.170+0000 7f67068d5700 1 -- start start 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f67068d5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67001094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f67068d5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6700060360 con 0x7f67000f5480 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67001094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67001094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:48506/0 (socket says 172.21.15.76:48506) 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fffff700 1 -- 172.21.15.76:0/3301670356 learned_addr learned my addr 172.21.15.76:0/3301670356 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fffff700 1 -- 172.21.15.76:0/3301670356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66f0005040 con 0x7f67000f5480 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fffff700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67001094b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f66f00089f0 tx=0x7f66f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66f0009320 con 0x7f67000f5480 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6700060560 con 0x7f67000f5480 2023-12-26T22:01:54.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f66f0013ec0 con 0x7f67000f5480 2023-12-26T22:01:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66f0014d30 con 0x7f67000f5480 2023-12-26T22:01:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.171+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6700109be0 con 0x7f67000f5480 2023-12-26T22:01:54.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.172+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f66f00095f0 con 0x7f67000f5480 2023-12-26T22:01:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.172+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6700045bb0 con 0x7f67000f5480 2023-12-26T22:01:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.172+0000 7f66fcff9700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f66e8031090 0x7f66e8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.172+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f66f00424a0 con 0x7f67000f5480 2023-12-26T22:01:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.173+0000 7f66ff7fe700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f66e8031090 0x7f66e8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:54.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.173+0000 7f66ff7fe700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f66e8031090 0x7f66e8033550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f66f40096a0 tx=0x7f66f4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:54.181 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.181+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f66f001d980 con 0x7f67000f5480 2023-12-26T22:01:54.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.386+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f670010a400 con 0x7f67000f5480 2023-12-26T22:01:54.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.387+0000 7f66fcff9700 1 -- 172.21.15.76:0/3301670356 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f66f001d380 con 0x7f67000f5480 2023-12-26T22:01:54.387 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:54.387 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f66e8031090 msgr2=0x7f66e8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f66e8031090 0x7f66e8033550 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f66f40096a0 tx=0x7f66f4006b40).stop 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 msgr2=0x7f67001094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67001094b0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f66f00089f0 tx=0x7f66f000b040).stop 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 shutdown_connections 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f66e8031090 0x7f66e8033550 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 --2- 172.21.15.76:0/3301670356 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67000f5480 0x7f67001094b0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:54.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 >> 172.21.15.76:0/3301670356 conn(0x7f67000f0be0 msgr2=0x7f67000f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 shutdown_connections 2023-12-26T22:01:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:54.389+0000 7f67068d5700 1 -- 172.21.15.76:0/3301670356 wait complete. 2023-12-26T22:01:54.390 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:55.148 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:53.372855+0000 mgr.smithi059.gwkiyc (mgr.14156) 156 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:55.148 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:54.122978+0000 mon.smithi059 (mon.0) 218 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-12-26T22:01:55.149 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:54.126430+0000 mon.smithi059 (mon.0) 219 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi059", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-12-26T22:01:55.149 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:54.127246+0000 mon.smithi059 (mon.0) 220 : audit [DBG] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-12-26T22:01:55.149 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:54.388792+0000 mon.smithi059 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.76:0/3301670356' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:55.771 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:55.771 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:01:54.120883+0000 mgr.smithi059.gwkiyc (mgr.14156) 157 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-761f1ff0-a439-11ee-95a5-87774f69a715-alertmanager.smithi059 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: /bin/podman: stderr Error: no such container ceph-761f1ff0-a439-11ee-95a5-87774f69a715-alertmanager.smithi059 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Deploy daemon alertmanager.smithi059 ... 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:57:32Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:58:10Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:58:47Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:00:01Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:00:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:01:16Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-12-26T22:01:56.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: main() 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: r = ctx.func(ctx) 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: return func(ctx) 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: raise RuntimeError('uid/gid not found') 2023-12-26T22:01:56.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: RuntimeError: uid/gid not found 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: yield (conn, connr) 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: code, '\n'.join(err))) 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-761f1ff0-a439-11ee-95a5-87774f69a715-alertmanager.smithi059 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: /bin/podman: stderr Error: no such container ceph-761f1ff0-a439-11ee-95a5-87774f69a715-alertmanager.smithi059 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Deploy daemon alertmanager.smithi059 ... 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:57:32Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:58:10Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:58:47Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:00:01Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:00:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:01:16Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: main() 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-12-26T22:01:56.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: r = ctx.func(ctx) 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: return func(ctx) 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: raise RuntimeError('uid/gid not found') 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: RuntimeError: uid/gid not found 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:01:54.121187+0000 mgr.smithi059.gwkiyc (mgr.14156) 158 : cephadm [ERR] Failed while placing alertmanager.smithi059 on smithi059: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-761f1ff0-a439-11ee-95a5-87774f69a715-alertmanager.smithi059 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: /bin/podman: stderr Error: no such container ceph-761f1ff0-a439-11ee-95a5-87774f69a715-alertmanager.smithi059 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Deploy daemon alertmanager.smithi059 ... 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-12-26T22:01:56.301 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:57:32Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:58:10Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T21:58:47Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --stop-signal=SIGTERM --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi059 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:00:01Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:00:38Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr time="2023-12-26T22:01:16Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:01:56.302 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: main() 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: r = ctx.func(ctx) 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: return func(ctx) 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4344, in command_deploy 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4275, in extract_uid_gid_monitoring 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-12-26T22:01:56.303 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2548, in extract_uid_gid 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: raise RuntimeError('uid/gid not found') 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: RuntimeError: uid/gid not found 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:01:54.128022+0000 mgr.smithi059.gwkiyc (mgr.14156) 159 : cephadm [INF] Deploying daemon crash.smithi059 on smithi059 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.467084+0000 mon.smithi059 (mon.0) 222 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.752456+0000 mon.smithi059 (mon.0) 223 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.812679+0000 mon.smithi059 (mon.0) 224 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.814708+0000 mon.smithi059 (mon.0) 225 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.815086+0000 mon.smithi059 (mon.0) 226 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-26T22:01:56.304 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:55 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.817754+0000 mon.smithi059 (mon.0) 227 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:01:56.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.954+0000 7f3560c68700 1 -- 172.21.15.76:0/3939763853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 msgr2=0x7f355c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:56.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.954+0000 7f3560c68700 1 --2- 172.21.15.76:0/3939763853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c0f69b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f3550004760 tx=0x7f3550009ad0).stop 2023-12-26T22:01:56.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.954+0000 7f3560c68700 1 -- 172.21.15.76:0/3939763853 shutdown_connections 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.954+0000 7f3560c68700 1 --2- 172.21.15.76:0/3939763853 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c0f69b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.954+0000 7f3560c68700 1 -- 172.21.15.76:0/3939763853 >> 172.21.15.76:0/3939763853 conn(0x7f355c0f1cd0 msgr2=0x7f355c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f3560c68700 1 -- 172.21.15.76:0/3939763853 shutdown_connections 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f3560c68700 1 -- 172.21.15.76:0/3939763853 wait complete. 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f3560c68700 1 Processor -- start 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f3560c68700 1 -- start start 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f3560c68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f3560c68700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f355c106cb0 con 0x7f355c0f6590 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f355a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f355a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52766/0 (socket says 172.21.15.76:52766) 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.955+0000 7f355a59c700 1 -- 172.21.15.76:0/443996507 learned_addr learned my addr 172.21.15.76:0/443996507 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f355a59c700 1 -- 172.21.15.76:0/443996507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3550009d70 con 0x7f355c0f6590 2023-12-26T22:01:56.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f355a59c700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c1067c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f3550004170 tx=0x7f3550009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:56.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f355000e040 con 0x7f355c0f6590 2023-12-26T22:01:56.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f355c106eb0 con 0x7f355c0f6590 2023-12-26T22:01:56.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3550008ef0 con 0x7f355c0f6590 2023-12-26T22:01:56.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3550014c90 con 0x7f355c0f6590 2023-12-26T22:01:56.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.956+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f355c107340 con 0x7f355c0f6590 2023-12-26T22:01:56.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.957+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3550014420 con 0x7f355c0f6590 2023-12-26T22:01:56.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.957+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f355c045bb0 con 0x7f355c0f6590 2023-12-26T22:01:56.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.957+0000 7f354affd700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3544031090 0x7f3544033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:56.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.957+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3550042320 con 0x7f355c0f6590 2023-12-26T22:01:56.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.957+0000 7f3559d9b700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3544031090 0x7f3544033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:56.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.958+0000 7f3559d9b700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3544031090 0x7f3544033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f354c0096a0 tx=0x7f354c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:56.962 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:56.962+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3550026300 con 0x7f355c0f6590 2023-12-26T22:01:57.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.170+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f355c108190 con 0x7f355c0f6590 2023-12-26T22:01:57.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.171+0000 7f354affd700 1 -- 172.21.15.76:0/443996507 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3550012b20 con 0x7f355c0f6590 2023-12-26T22:01:57.171 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:57.171 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3544031090 msgr2=0x7f3544033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3544031090 0x7f3544033550 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f354c0096a0 tx=0x7f354c006b40).stop 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 msgr2=0x7f355c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c1067c0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f3550004170 tx=0x7f3550009ad0).stop 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 shutdown_connections 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3544031090 0x7f3544033550 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 --2- 172.21.15.76:0/443996507 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f355c0f6590 0x7f355c1067c0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 >> 172.21.15.76:0/443996507 conn(0x7f355c0f1cd0 msgr2=0x7f355c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:57.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 shutdown_connections 2023-12-26T22:01:57.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:57.173+0000 7f3560c68700 1 -- 172.21.15.76:0/443996507 wait complete. 2023-12-26T22:01:57.174 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:01:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:55.373071+0000 mgr.smithi059.gwkiyc (mgr.14156) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:55.815415+0000 mgr.smithi059.gwkiyc (mgr.14156) 161 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-12-26T22:01:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:01:55.820981+0000 mgr.smithi059.gwkiyc (mgr.14156) 162 : cephadm [INF] Deploying daemon grafana.smithi059 on smithi059 2023-12-26T22:01:58.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:57 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:57.172733+0000 mon.smithi059 (mon.0) 228 : audit [DBG] from='client.? 172.21.15.76:0/443996507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:01:58.552 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:01:58.552 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:01:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:01:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:57.373415+0000 mgr.smithi059.gwkiyc (mgr.14156) 163 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 -- 172.21.15.76:0/1006719663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 msgr2=0x7f2c780f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 --2- 172.21.15.76:0/1006719663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c780f58a0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f2c6c009fb0 tx=0x7f2c6c009b30).stop 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 -- 172.21.15.76:0/1006719663 shutdown_connections 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 --2- 172.21.15.76:0/1006719663 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c780f58a0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 -- 172.21.15.76:0/1006719663 >> 172.21.15.76:0/1006719663 conn(0x7f2c780f0be0 msgr2=0x7f2c780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 -- 172.21.15.76:0/1006719663 shutdown_connections 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 -- 172.21.15.76:0/1006719663 wait complete. 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 Processor -- start 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.720+0000 7f2c7f20f700 1 -- start start 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7f20f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c781056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:59.722 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7f20f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c78105be0 con 0x7f2c780f5480 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7cfab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c781056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7cfab700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c781056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52780/0 (socket says 172.21.15.76:52780) 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7cfab700 1 -- 172.21.15.76:0/2216102236 learned_addr learned my addr 172.21.15.76:0/2216102236 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7cfab700 1 -- 172.21.15.76:0/2216102236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c6c005040 con 0x7f2c780f5480 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.721+0000 7f2c7cfab700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c781056f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2c6c0089f0 tx=0x7f2c6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6c009320 con 0x7f2c780f5480 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c78105de0 con 0x7f2c780f5480 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2c6c013ec0 con 0x7f2c780f5480 2023-12-26T22:01:59.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2c6c014d30 con 0x7f2c780f5480 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c78106270 con 0x7f2c780f5480 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2c6c014420 con 0x7f2c780f5480 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c757fa700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2c68031090 0x7f2c68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.722+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c78045bb0 con 0x7f2c780f5480 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.723+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2c6c042400 con 0x7f2c780f5480 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.723+0000 7f2c77fff700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2c68031090 0x7f2c68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:01:59.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.723+0000 7f2c77fff700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2c68031090 0x7f2c68033550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2c640096a0 tx=0x7f2c64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:01:59.727 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.727+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2c6c014750 con 0x7f2c780f5480 2023-12-26T22:01:59.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.936+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2c78107070 con 0x7f2c780f5480 2023-12-26T22:01:59.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.937+0000 7f2c757fa700 1 -- 172.21.15.76:0/2216102236 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2c6c01d380 con 0x7f2c780f5480 2023-12-26T22:01:59.937 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:01:59.937 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:01:59.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2c68031090 msgr2=0x7f2c68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:59.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2c68031090 0x7f2c68033550 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f2c640096a0 tx=0x7f2c64006b40).stop 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 msgr2=0x7f2c781056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c781056f0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2c6c0089f0 tx=0x7f2c6c00b040).stop 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 shutdown_connections 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2c68031090 0x7f2c68033550 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 --2- 172.21.15.76:0/2216102236 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2c780f5480 0x7f2c781056f0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 >> 172.21.15.76:0/2216102236 conn(0x7f2c780f0be0 msgr2=0x7f2c780f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 shutdown_connections 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:01:59.938+0000 7f2c7f20f700 1 -- 172.21.15.76:0/2216102236 wait complete. 2023-12-26T22:01:59.939 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:01:59.373863+0000 mgr.smithi059.gwkiyc (mgr.14156) 164 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:01:59.938292+0000 mon.smithi059 (mon.0) 229 : audit [DBG] from='client.? 172.21.15.76:0/2216102236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:00.470070+0000 mon.smithi059 (mon.0) 230 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:02:01.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:01.319 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:02.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 -- 172.21.15.76:0/2816243480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 msgr2=0x7fe8e80f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 --2- 172.21.15.76:0/2816243480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e80f56f0 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fe8dc009fb0 tx=0x7fe8dc009b30).stop 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 -- 172.21.15.76:0/2816243480 shutdown_connections 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 --2- 172.21.15.76:0/2816243480 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e80f56f0 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 -- 172.21.15.76:0/2816243480 >> 172.21.15.76:0/2816243480 conn(0x7fe8e80f0a10 msgr2=0x7fe8e80f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 -- 172.21.15.76:0/2816243480 shutdown_connections 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 -- 172.21.15.76:0/2816243480 wait complete. 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.503+0000 7fe8edb74700 1 Processor -- start 2023-12-26T22:02:02.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8edb74700 1 -- start start 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8edb74700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e8105540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8edb74700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe8e8105a30 con 0x7fe8e80f52d0 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8ecb72700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e8105540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8ecb72700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e8105540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52792/0 (socket says 172.21.15.76:52792) 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8ecb72700 1 -- 172.21.15.76:0/2707745640 learned_addr learned my addr 172.21.15.76:0/2707745640 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.504+0000 7fe8ecb72700 1 -- 172.21.15.76:0/2707745640 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe8dc005040 con 0x7fe8e80f52d0 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.505+0000 7fe8ecb72700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e8105540 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fe8dc0089f0 tx=0x7fe8dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:02.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.505+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8dc009320 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.505+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe8e8105c30 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.505+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe8dc013ec0 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.505+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe8dc014d30 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.505+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8e81060c0 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.506+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe8dc0263a0 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.506+0000 7fe8e57fa700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe8d8031090 0x7fe8d8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.506+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe8dc0423a0 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.506+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe8e8045bb0 con 0x7fe8e80f52d0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.506+0000 7fe8e7fff700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe8d8031090 0x7fe8d8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:02.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.506+0000 7fe8e7fff700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe8d8031090 0x7fe8d8033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe8d40096a0 tx=0x7fe8d4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:02.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.511+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe8dc012920 con 0x7fe8e80f52d0 2023-12-26T22:02:02.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.718+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8e8106d70 con 0x7fe8e80f52d0 2023-12-26T22:02:02.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.718+0000 7fe8e57fa700 1 -- 172.21.15.76:0/2707745640 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe8dc012330 con 0x7fe8e80f52d0 2023-12-26T22:02:02.718 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:02.719 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe8d8031090 msgr2=0x7fe8d8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe8d8031090 0x7fe8d8033550 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe8d40096a0 tx=0x7fe8d4006b40).stop 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 msgr2=0x7fe8e8105540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e8105540 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fe8dc0089f0 tx=0x7fe8dc00b040).stop 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 shutdown_connections 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe8d8031090 0x7fe8d8033550 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 --2- 172.21.15.76:0/2707745640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe8e80f52d0 0x7fe8e8105540 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 >> 172.21.15.76:0/2707745640 conn(0x7fe8e80f0a10 msgr2=0x7fe8e80f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:02.720 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 shutdown_connections 2023-12-26T22:02:02.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:02.720+0000 7fe8edb74700 1 -- 172.21.15.76:0/2707745640 wait complete. 2023-12-26T22:02:02.721 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:01.374199+0000 mgr.smithi059.gwkiyc (mgr.14156) 165 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:02.719988+0000 mon.smithi059 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.76:0/2707745640' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:04.115 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:04.116 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:03.374549+0000 mgr.smithi059.gwkiyc (mgr.14156) 166 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:05.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- 172.21.15.76:0/1955143100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 msgr2=0x7f61780f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:05.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 --2- 172.21.15.76:0/1955143100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61780f69b0 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f6174004760 tx=0x7f6174009ad0).stop 2023-12-26T22:02:05.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- 172.21.15.76:0/1955143100 shutdown_connections 2023-12-26T22:02:05.335 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 --2- 172.21.15.76:0/1955143100 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61780f69b0 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- 172.21.15.76:0/1955143100 >> 172.21.15.76:0/1955143100 conn(0x7f61780f1cd0 msgr2=0x7f61780f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- 172.21.15.76:0/1955143100 shutdown_connections 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- 172.21.15.76:0/1955143100 wait complete. 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 Processor -- start 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- start start 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61781067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.334+0000 7f6180220700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6178106cb0 con 0x7f61780f6590 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f617dfbc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61781067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f617dfbc700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61781067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52802/0 (socket says 172.21.15.76:52802) 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f617dfbc700 1 -- 172.21.15.76:0/3701529121 learned_addr learned my addr 172.21.15.76:0/3701529121 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f617dfbc700 1 -- 172.21.15.76:0/3701529121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6174009d70 con 0x7f61780f6590 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f617dfbc700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61781067c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f6174004170 tx=0x7f6174009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f617400e040 con 0x7f61780f6590 2023-12-26T22:02:05.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6174008ef0 con 0x7f61780f6590 2023-12-26T22:02:05.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6178106eb0 con 0x7f61780f6590 2023-12-26T22:02:05.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6174014c90 con 0x7f61780f6590 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.335+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6178107340 con 0x7f61780f6590 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.336+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6174014420 con 0x7f61780f6590 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.336+0000 7f616e7fc700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6164031090 0x7f6164033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.336+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6174042240 con 0x7f61780f6590 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.336+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6178045bb0 con 0x7f61780f6590 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.336+0000 7f617d7bb700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6164031090 0x7f6164033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:05.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.337+0000 7f617d7bb700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6164031090 0x7f6164033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f61680096a0 tx=0x7f6168006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:05.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.341+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f6174012d00 con 0x7f61780f6590 2023-12-26T22:02:05.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.546+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61780fc170 con 0x7f61780f6590 2023-12-26T22:02:05.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.546+0000 7f616e7fc700 1 -- 172.21.15.76:0/3701529121 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f6174012b20 con 0x7f61780f6590 2023-12-26T22:02:05.547 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:05.547 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:05.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6164031090 msgr2=0x7f6164033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:05.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6164031090 0x7f6164033550 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f61680096a0 tx=0x7f6168006b40).stop 2023-12-26T22:02:05.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 msgr2=0x7f61781067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:05.548 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61781067c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f6174004170 tx=0x7f6174009ad0).stop 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 shutdown_connections 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6164031090 0x7f6164033550 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 --2- 172.21.15.76:0/3701529121 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f61780f6590 0x7f61781067c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 >> 172.21.15.76:0/3701529121 conn(0x7f61780f1cd0 msgr2=0x7f61780f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 shutdown_connections 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:05.548+0000 7f6180220700 1 -- 172.21.15.76:0/3701529121 wait complete. 2023-12-26T22:02:05.549 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:06.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:05 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:05.548143+0000 mon.smithi059 (mon.0) 232 : audit [DBG] from='client.? 172.21.15.76:0/3701529121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:06.944 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:06.945 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:05.374804+0000 mgr.smithi059.gwkiyc (mgr.14156) 167 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2874628096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 msgr2=0x7f3d240f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 --2- 172.21.15.76:0/2874628096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d240f67f0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f3d20009fb0 tx=0x7f3d20009b30).stop 2023-12-26T22:02:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2874628096 shutdown_connections 2023-12-26T22:02:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 --2- 172.21.15.76:0/2874628096 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d240f67f0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:08.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2874628096 >> 172.21.15.76:0/2874628096 conn(0x7f3d240f1b10 msgr2=0x7f3d240f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2874628096 shutdown_connections 2023-12-26T22:02:08.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.160+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2874628096 wait complete. 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d2a3f4700 1 Processor -- start 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d2a3f4700 1 -- start start 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d2a3f4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d24106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d2a3f4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d24106b30 con 0x7f3d240f63d0 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d293f2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d24106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d293f2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d24106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:36344/0 (socket says 172.21.15.76:36344) 2023-12-26T22:02:08.162 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d293f2700 1 -- 172.21.15.76:0/2866665349 learned_addr learned my addr 172.21.15.76:0/2866665349 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.161+0000 7f3d293f2700 1 -- 172.21.15.76:0/2866665349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d20005040 con 0x7f3d240f63d0 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.162+0000 7f3d293f2700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d24106640 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f3d200089f0 tx=0x7f3d2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.162+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d20009320 con 0x7f3d240f63d0 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.162+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3d20013ec0 con 0x7f3d240f63d0 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.162+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d20014d30 con 0x7f3d240f63d0 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.162+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d24106d30 con 0x7f3d240f63d0 2023-12-26T22:02:08.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.162+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d241071c0 con 0x7f3d240f63d0 2023-12-26T22:02:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.163+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3d200095f0 con 0x7f3d240f63d0 2023-12-26T22:02:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.163+0000 7f3d19ffb700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3d100310e0 0x7f3d100335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.163+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d24045bb0 con 0x7f3d240f63d0 2023-12-26T22:02:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.163+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3d20042490 con 0x7f3d240f63d0 2023-12-26T22:02:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.163+0000 7f3d28bf1700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3d100310e0 0x7f3d100335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:08.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.164+0000 7f3d28bf1700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3d100310e0 0x7f3d100335a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3d140096a0 tx=0x7f3d14006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:08.168 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.168+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3d2001d7f0 con 0x7f3d240f63d0 2023-12-26T22:02:08.373 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.373+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d24108010 con 0x7f3d240f63d0 2023-12-26T22:02:08.373 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.374+0000 7f3d19ffb700 1 -- 172.21.15.76:0/2866665349 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3d2000fe40 con 0x7f3d240f63d0 2023-12-26T22:02:08.374 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:08.374 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:08.375 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3d100310e0 msgr2=0x7f3d100335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:08.375 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3d100310e0 0x7f3d100335a0 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f3d140096a0 tx=0x7f3d14006b40).stop 2023-12-26T22:02:08.375 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 msgr2=0x7f3d24106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d24106640 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f3d200089f0 tx=0x7f3d2000b040).stop 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 shutdown_connections 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3d100310e0 0x7f3d100335a0 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 --2- 172.21.15.76:0/2866665349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3d240f63d0 0x7f3d24106640 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 >> 172.21.15.76:0/2866665349 conn(0x7f3d240f1b10 msgr2=0x7f3d240f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 shutdown_connections 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:08.375+0000 7f3d2a3f4700 1 -- 172.21.15.76:0/2866665349 wait complete. 2023-12-26T22:02:08.376 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:07.375070+0000 mgr.smithi059.gwkiyc (mgr.14156) 168 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:08.375207+0000 mon.smithi059 (mon.0) 233 : audit [DBG] from='client.? 172.21.15.76:0/2866665349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:09.773 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:09.774 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 -- 172.21.15.76:0/796469259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 msgr2=0x7f7aec0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 --2- 172.21.15.76:0/796469259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec0f56f0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f7adc009fb0 tx=0x7f7adc009b30).stop 2023-12-26T22:02:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 -- 172.21.15.76:0/796469259 shutdown_connections 2023-12-26T22:02:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 --2- 172.21.15.76:0/796469259 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec0f56f0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 -- 172.21.15.76:0/796469259 >> 172.21.15.76:0/796469259 conn(0x7f7aec0f0a30 msgr2=0x7f7aec0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:10.952 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 -- 172.21.15.76:0/796469259 shutdown_connections 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.950+0000 7f7af0bc5700 1 -- 172.21.15.76:0/796469259 wait complete. 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7af0bc5700 1 Processor -- start 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7af0bc5700 1 -- start start 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7af0bc5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7af0bc5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aec106af0 con 0x7f7aec0f52d0 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7aeb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7aeb7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:36366/0 (socket says 172.21.15.76:36366) 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.951+0000 7f7aeb7fe700 1 -- 172.21.15.76:0/1045094098 learned_addr learned my addr 172.21.15.76:0/1045094098 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7aeb7fe700 1 -- 172.21.15.76:0/1045094098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7adc005040 con 0x7f7aec0f52d0 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7aeb7fe700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec106600 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f7adc000c00 tx=0x7f7adc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7adc0095f0 con 0x7f7aec0f52d0 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7adc007e50 con 0x7f7aec0f52d0 2023-12-26T22:02:10.953 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aec106cf0 con 0x7f7aec0f52d0 2023-12-26T22:02:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7adc014e70 con 0x7f7aec0f52d0 2023-12-26T22:02:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.952+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aec107180 con 0x7f7aec0f52d0 2023-12-26T22:02:10.954 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.953+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7adc025070 con 0x7f7aec0f52d0 2023-12-26T22:02:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.953+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aec045bb0 con 0x7f7aec0f52d0 2023-12-26T22:02:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.953+0000 7f7ad3fff700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7ad4031090 0x7f7ad4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.953+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7adc041e00 con 0x7f7aec0f52d0 2023-12-26T22:02:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.953+0000 7f7aeaffd700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7ad4031090 0x7f7ad4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:10.955 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.954+0000 7f7aeaffd700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7ad4031090 0x7f7ad4033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f7ae00096a0 tx=0x7f7ae0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:10.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:10.957+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7adc00f4e0 con 0x7f7aec0f52d0 2023-12-26T22:02:11.164 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.164+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7aec107f90 con 0x7f7aec0f52d0 2023-12-26T22:02:11.165 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.164+0000 7f7ad3fff700 1 -- 172.21.15.76:0/1045094098 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f7adc00f300 con 0x7f7aec0f52d0 2023-12-26T22:02:11.165 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:11.165 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7ad4031090 msgr2=0x7f7ad4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:11.166 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7ad4031090 0x7f7ad4033550 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f7ae00096a0 tx=0x7f7ae0006b40).stop 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 msgr2=0x7f7aec106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec106600 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f7adc000c00 tx=0x7f7adc00b040).stop 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 shutdown_connections 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f7ad4031090 0x7f7ad4033550 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 --2- 172.21.15.76:0/1045094098 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f7aec0f52d0 0x7f7aec106600 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 >> 172.21.15.76:0/1045094098 conn(0x7f7aec0f0a30 msgr2=0x7f7aec05fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 shutdown_connections 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:11.166+0000 7f7af0bc5700 1 -- 172.21.15.76:0/1045094098 wait complete. 2023-12-26T22:02:11.167 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:09.375344+0000 mgr.smithi059.gwkiyc (mgr.14156) 169 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:12.194 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:11 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:11.166127+0000 mon.smithi059 (mon.0) 234 : audit [DBG] from='client.? 172.21.15.76:0/1045094098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:12.504 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:12.504 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:13.070 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:11.375559+0000 mgr.smithi059.gwkiyc (mgr.14156) 170 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:13.737 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 -- 172.21.15.76:0/1517605015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 msgr2=0x7fc1e00f91c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:13.737 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 --2- 172.21.15.76:0/1517605015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e00f91c0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc1dc009fb0 tx=0x7fc1dc009b30).stop 2023-12-26T22:02:13.737 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 -- 172.21.15.76:0/1517605015 shutdown_connections 2023-12-26T22:02:13.737 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 --2- 172.21.15.76:0/1517605015 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e00f91c0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 -- 172.21.15.76:0/1517605015 >> 172.21.15.76:0/1517605015 conn(0x7fc1e00f0be0 msgr2=0x7fc1e00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 -- 172.21.15.76:0/1517605015 shutdown_connections 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.737+0000 7fc1e759c700 1 -- 172.21.15.76:0/1517605015 wait complete. 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e759c700 1 Processor -- start 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e759c700 1 -- start start 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e759c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e0106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e759c700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1e0106c70 con 0x7fc1e0060300 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e5338700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e0106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e5338700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e0106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:36384/0 (socket says 172.21.15.76:36384) 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e5338700 1 -- 172.21.15.76:0/2173714499 learned_addr learned my addr 172.21.15.76:0/2173714499 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e5338700 1 -- 172.21.15.76:0/2173714499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1dc005040 con 0x7fc1e0060300 2023-12-26T22:02:13.738 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.738+0000 7fc1e5338700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e0106780 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc1dc009f80 tx=0x7fc1dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.739+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1dc009320 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.739+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fc1dc013680 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.739+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1e0106e70 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.739+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc1dc01eda0 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.739+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc1e0107300 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.739+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fc1dc01e420 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.740+0000 7fc1d5ffb700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc1cc031090 0x7fc1cc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.740+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fc1dc0423f0 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.740+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1e0045bb0 con 0x7fc1e0060300 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.740+0000 7fc1e4b37700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc1cc031090 0x7fc1cc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:13.740 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.740+0000 7fc1e4b37700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc1cc031090 0x7fc1cc033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc1d00096a0 tx=0x7fc1d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:13.743 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.744+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fc1dc01e700 con 0x7fc1e0060300 2023-12-26T22:02:13.948 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.949+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc1e0108040 con 0x7fc1e0060300 2023-12-26T22:02:13.949 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.950+0000 7fc1d5ffb700 1 -- 172.21.15.76:0/2173714499 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fc1dc01cb10 con 0x7fc1e0060300 2023-12-26T22:02:13.949 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:13.949 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc1cc031090 msgr2=0x7fc1cc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc1cc031090 0x7fc1cc033550 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc1d00096a0 tx=0x7fc1d0006b40).stop 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 msgr2=0x7fc1e0106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e0106780 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc1dc009f80 tx=0x7fc1dc00b040).stop 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 shutdown_connections 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc1cc031090 0x7fc1cc033550 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 --2- 172.21.15.76:0/2173714499 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fc1e0060300 0x7fc1e0106780 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 >> 172.21.15.76:0/2173714499 conn(0x7fc1e00f0be0 msgr2=0x7fc1e00f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 shutdown_connections 2023-12-26T22:02:13.951 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:13.952+0000 7fc1e759c700 1 -- 172.21.15.76:0/2173714499 wait complete. 2023-12-26T22:02:13.952 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:14.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:13 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:12.932849+0000 mon.smithi059 (mon.0) 235 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:02:14.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:13 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:02:12.934686+0000 mgr.smithi059.gwkiyc (mgr.14156) 171 : cephadm [INF] Deploying daemon node-exporter.smithi059 on smithi059 2023-12-26T22:02:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:13.376317+0000 mgr.smithi059.gwkiyc (mgr.14156) 172 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:13.950732+0000 mon.smithi059 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.76:0/2173714499' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:15.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:15.322 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:16.543 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 -- 172.21.15.76:0/1067556380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 msgr2=0x7f82b40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 --2- 172.21.15.76:0/1067556380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b40f69b0 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f829c009fb0 tx=0x7f829c009b30).stop 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 -- 172.21.15.76:0/1067556380 shutdown_connections 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 --2- 172.21.15.76:0/1067556380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b40f69b0 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 -- 172.21.15.76:0/1067556380 >> 172.21.15.76:0/1067556380 conn(0x7f82b40f1cf0 msgr2=0x7f82b40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 -- 172.21.15.76:0/1067556380 shutdown_connections 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 -- 172.21.15.76:0/1067556380 wait complete. 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.543+0000 7f82ba33d700 1 Processor -- start 2023-12-26T22:02:16.544 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82ba33d700 1 -- start start 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82ba33d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b4106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82ba33d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f82b4106c70 con 0x7f82b40f6590 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82b3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b4106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82b3fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b4106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:37806/0 (socket says 172.21.15.76:37806) 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82b3fff700 1 -- 172.21.15.76:0/3636198162 learned_addr learned my addr 172.21.15.76:0/3636198162 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82b3fff700 1 -- 172.21.15.76:0/3636198162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f829c005040 con 0x7f82b40f6590 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82b3fff700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b4106780 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f829c0089f0 tx=0x7f829c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.544+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f829c009320 con 0x7f82b40f6590 2023-12-26T22:02:16.545 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.545+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f82b4106e70 con 0x7f82b40f6590 2023-12-26T22:02:16.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.545+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f829c013ec0 con 0x7f82b40f6590 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.545+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f829c014d30 con 0x7f82b40f6590 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.545+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82b4107300 con 0x7f82b40f6590 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.545+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f829c014420 con 0x7f82b40f6590 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.546+0000 7f82b0ff9700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f82a0031090 0x7f82a0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.546+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f82b4045bb0 con 0x7f82b40f6590 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.546+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f829c0423e0 con 0x7f82b40f6590 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.546+0000 7f82b37fe700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f82a0031090 0x7f82a0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:16.547 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.546+0000 7f82b37fe700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f82a0031090 0x7f82a0033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f82a40096a0 tx=0x7f82a4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:16.549 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.551+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f829c014750 con 0x7f82b40f6590 2023-12-26T22:02:16.757 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.758+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f82b4108100 con 0x7f82b40f6590 2023-12-26T22:02:16.758 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.759+0000 7f82b0ff9700 1 -- 172.21.15.76:0/3636198162 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f829c01d7f0 con 0x7f82b40f6590 2023-12-26T22:02:16.758 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:16.758 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:16.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f82a0031090 msgr2=0x7f82a0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:16.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f82a0031090 0x7f82a0033550 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f82a40096a0 tx=0x7f82a4006b40).stop 2023-12-26T22:02:16.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 msgr2=0x7f82b4106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:16.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b4106780 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f829c0089f0 tx=0x7f829c00b040).stop 2023-12-26T22:02:16.760 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 shutdown_connections 2023-12-26T22:02:16.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f82a0031090 0x7f82a0033550 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:16.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 --2- 172.21.15.76:0/3636198162 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f82b40f6590 0x7f82b4106780 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:16.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 >> 172.21.15.76:0/3636198162 conn(0x7f82b40f1cf0 msgr2=0x7f82b40f27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:16.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 shutdown_connections 2023-12-26T22:02:16.761 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:16.760+0000 7f82ba33d700 1 -- 172.21.15.76:0/3636198162 wait complete. 2023-12-26T22:02:16.761 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:16.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:15.376882+0000 mgr.smithi059.gwkiyc (mgr.14156) 173 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:16.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:15.473400+0000 mon.smithi059 (mon.0) 237 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:02:17.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:17 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:16.759421+0000 mon.smithi059 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.76:0/3636198162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:18.150 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:18.150 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:19.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:17.377187+0000 mgr.smithi059.gwkiyc (mgr.14156) 174 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.401+0000 7f632359e700 1 -- 172.21.15.76:0/1950959855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 msgr2=0x7f63240f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.401+0000 7f632359e700 1 --2- 172.21.15.76:0/1950959855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f63240f6800 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f6318009fb0 tx=0x7f6318009b30).stop 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.401+0000 7f632359e700 1 -- 172.21.15.76:0/1950959855 shutdown_connections 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.401+0000 7f632359e700 1 --2- 172.21.15.76:0/1950959855 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f63240f6800 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.401+0000 7f632359e700 1 -- 172.21.15.76:0/1950959855 >> 172.21.15.76:0/1950959855 conn(0x7f63240f1b40 msgr2=0x7f63240f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632359e700 1 -- 172.21.15.76:0/1950959855 shutdown_connections 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632359e700 1 -- 172.21.15.76:0/1950959855 wait complete. 2023-12-26T22:02:19.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632359e700 1 Processor -- start 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632359e700 1 -- start start 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632359e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f6324106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632359e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6324106b10 con 0x7f63240f63e0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f6324106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f6324106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:37820/0 (socket says 172.21.15.76:37820) 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.402+0000 7f632259c700 1 -- 172.21.15.76:0/3253849536 learned_addr learned my addr 172.21.15.76:0/3253849536 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f632259c700 1 -- 172.21.15.76:0/3253849536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6318005040 con 0x7f63240f63e0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f632259c700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f6324106620 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6318000c00 tx=0x7f631800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63180095f0 con 0x7f63240f63e0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6324106d10 con 0x7f63240f63e0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f6318007e50 con 0x7f63240f63e0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6318014e70 con 0x7f63240f63e0 2023-12-26T22:02:19.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.403+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63241071a0 con 0x7f63240f63e0 2023-12-26T22:02:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.404+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f6318025070 con 0x7f63240f63e0 2023-12-26T22:02:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.404+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6324045bb0 con 0x7f63240f63e0 2023-12-26T22:02:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.404+0000 7f6312ffd700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f630c031090 0x7f630c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:19.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.404+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f6318041e00 con 0x7f63240f63e0 2023-12-26T22:02:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.404+0000 7f6321d9b700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f630c031090 0x7f630c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:19.405 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.405+0000 7f6321d9b700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f630c031090 0x7f630c033550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f63140096a0 tx=0x7f6314006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:19.408 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.409+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f631800f4e0 con 0x7f63240f63e0 2023-12-26T22:02:19.613 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.615+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6324107fa0 con 0x7f63240f63e0 2023-12-26T22:02:19.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.615+0000 7f6312ffd700 1 -- 172.21.15.76:0/3253849536 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f631800f300 con 0x7f63240f63e0 2023-12-26T22:02:19.614 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:19.614 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.616+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f630c031090 msgr2=0x7f630c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.616+0000 7f632359e700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f630c031090 0x7f630c033550 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f63140096a0 tx=0x7f6314006b40).stop 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.616+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 msgr2=0x7f6324106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.616+0000 7f632359e700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f6324106620 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f6318000c00 tx=0x7f631800b040).stop 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.617+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 shutdown_connections 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.617+0000 7f632359e700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f630c031090 0x7f630c033550 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.617+0000 7f632359e700 1 --2- 172.21.15.76:0/3253849536 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f63240f63e0 0x7f6324106620 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.617+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 >> 172.21.15.76:0/3253849536 conn(0x7f63240f1b40 msgr2=0x7f63240f26a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.617+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 shutdown_connections 2023-12-26T22:02:19.616 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:19.617+0000 7f632359e700 1 -- 172.21.15.76:0/3253849536 wait complete. 2023-12-26T22:02:19.617 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:20.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:19 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:19.615791+0000 mon.smithi059 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.76:0/3253849536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:21.009 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:21.009 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:21.044 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:19.377580+0000 mgr.smithi059.gwkiyc (mgr.14156) 175 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/1802152309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 msgr2=0x7fb9dc0f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 --2- 172.21.15.76:0/1802152309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc0f58d0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7fb9d8009fb0 tx=0x7fb9d8009b30).stop 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/1802152309 shutdown_connections 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 --2- 172.21.15.76:0/1802152309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc0f58d0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/1802152309 >> 172.21.15.76:0/1802152309 conn(0x7fb9dc0f0c10 msgr2=0x7fb9dc0f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/1802152309 shutdown_connections 2023-12-26T22:02:22.244 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.244+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/1802152309 wait complete. 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e3fc5700 1 Processor -- start 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e3fc5700 1 -- start start 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e3fc5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc1067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e3fc5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9dc106ce0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e1d61700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc1067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e1d61700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc1067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:37834/0 (socket says 172.21.15.76:37834) 2023-12-26T22:02:22.245 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e1d61700 1 -- 172.21.15.76:0/3334690240 learned_addr learned my addr 172.21.15.76:0/3334690240 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:22.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e1d61700 1 -- 172.21.15.76:0/3334690240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb9d8005040 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.246 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.245+0000 7fb9e1d61700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc1067f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fb9d80089f0 tx=0x7fb9d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.246+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d8009320 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.246+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb9dc106ee0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.246+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb9d8013ec0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.246+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb9d8014d30 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.246+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb9dc107370 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.247+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb9d80095f0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.247+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb9dc045bb0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.247+0000 7fb9d27fc700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb9c8031090 0x7fb9c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.247+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb9d8042440 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.247+0000 7fb9e1560700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb9c8031090 0x7fb9c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:22.247 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.247+0000 7fb9e1560700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb9c8031090 0x7fb9c8033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb9cc0096a0 tx=0x7fb9cc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:22.250 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.252+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb9d801d7f0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.459+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb9dc0fafa0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.459 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.460+0000 7fb9d27fc700 1 -- 172.21.15.76:0/3334690240 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb9d801d7f0 con 0x7fb9dc0f54b0 2023-12-26T22:02:22.459 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:22.459 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:22.460 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.461+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb9c8031090 msgr2=0x7fb9c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:22.460 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.461+0000 7fb9e3fc5700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb9c8031090 0x7fb9c8033550 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fb9cc0096a0 tx=0x7fb9cc006b40).stop 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.461+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 msgr2=0x7fb9dc1067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.461+0000 7fb9e3fc5700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc1067f0 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fb9d80089f0 tx=0x7fb9d800b040).stop 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.461+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 shutdown_connections 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.461+0000 7fb9e3fc5700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb9c8031090 0x7fb9c8033550 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.462+0000 7fb9e3fc5700 1 --2- 172.21.15.76:0/3334690240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb9dc0f54b0 0x7fb9dc1067f0 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.462+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 >> 172.21.15.76:0/3334690240 conn(0x7fb9dc0f0c10 msgr2=0x7fb9dc0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.462+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 shutdown_connections 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:22.462+0000 7fb9e3fc5700 1 -- 172.21.15.76:0/3334690240 wait complete. 2023-12-26T22:02:22.461 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:21.377884+0000 mgr.smithi059.gwkiyc (mgr.14156) 176 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:22.460501+0000 mon.smithi059 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.76:0/3334690240' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:23.859 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:23.859 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 -- 172.21.15.76:0/1068391446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 msgr2=0x7f4a040f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 --2- 172.21.15.76:0/1068391446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a040f58a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f49f4009fb0 tx=0x7f49f4009b30).stop 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 -- 172.21.15.76:0/1068391446 shutdown_connections 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 --2- 172.21.15.76:0/1068391446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a040f58a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 -- 172.21.15.76:0/1068391446 >> 172.21.15.76:0/1068391446 conn(0x7f4a040f0be0 msgr2=0x7f4a040f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 -- 172.21.15.76:0/1068391446 shutdown_connections 2023-12-26T22:02:25.080 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.080+0000 7f4a0a090700 1 -- 172.21.15.76:0/1068391446 wait complete. 2023-12-26T22:02:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.081+0000 7f4a0a090700 1 Processor -- start 2023-12-26T22:02:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.081+0000 7f4a0a090700 1 -- start start 2023-12-26T22:02:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.081+0000 7f4a0a090700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a041067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:25.081 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.082+0000 7f4a0a090700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a04106cd0 con 0x7f4a040f5480 2023-12-26T22:02:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.082+0000 7f4a037fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a041067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.082+0000 7f4a037fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a041067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:37858/0 (socket says 172.21.15.76:37858) 2023-12-26T22:02:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.082+0000 7f4a037fe700 1 -- 172.21.15.76:0/540340205 learned_addr learned my addr 172.21.15.76:0/540340205 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.082+0000 7f4a037fe700 1 -- 172.21.15.76:0/540340205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49f4005040 con 0x7f4a040f5480 2023-12-26T22:02:25.082 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.082+0000 7f4a037fe700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a041067e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f49f40089f0 tx=0x7f49f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.083+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49f4009320 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.083+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f49f4013ec0 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.083+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a04106ed0 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.083+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f49f4014d30 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.083+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a04107360 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.083+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f49f40263a0 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.084+0000 7f49ebfff700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49ec031090 0x7f49ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.084+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a04045bb0 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.084+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f49f4042450 con 0x7f4a040f5480 2023-12-26T22:02:25.083 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.084+0000 7f4a02ffd700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49ec031090 0x7f49ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:25.084 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.084+0000 7f4a02ffd700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49ec031090 0x7f49ec033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f49f80096a0 tx=0x7f49f8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:25.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.089+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f49f40266d0 con 0x7f4a040f5480 2023-12-26T22:02:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:23.378300+0000 mgr.smithi059.gwkiyc (mgr.14156) 177 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:25.295 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.294+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a04108160 con 0x7f4a040f5480 2023-12-26T22:02:25.295 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.295+0000 7f49ebfff700 1 -- 172.21.15.76:0/540340205 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f49f402b020 con 0x7f4a040f5480 2023-12-26T22:02:25.296 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:25.296 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:25.297 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49ec031090 msgr2=0x7f49ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:25.297 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49ec031090 0x7f49ec033550 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f49f80096a0 tx=0x7f49f8006b40).stop 2023-12-26T22:02:25.297 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 msgr2=0x7f4a041067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:25.297 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a041067e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f49f40089f0 tx=0x7f49f400b040).stop 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 shutdown_connections 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f49ec031090 0x7f49ec033550 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 --2- 172.21.15.76:0/540340205 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a040f5480 0x7f4a041067e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 >> 172.21.15.76:0/540340205 conn(0x7f4a040f0be0 msgr2=0x7f4a0405ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 shutdown_connections 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:25.296+0000 7f4a0a090700 1 -- 172.21.15.76:0/540340205 wait complete. 2023-12-26T22:02:25.298 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:26.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:25.295224+0000 mon.smithi059 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.76:0/540340205' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:26.684 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:26.684 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:27.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:25.378612+0000 mgr.smithi059.gwkiyc (mgr.14156) 178 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.865+0000 7f71e4b89700 1 -- 172.21.15.76:0/1747270801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 msgr2=0x7f71e00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.865+0000 7f71e4b89700 1 --2- 172.21.15.76:0/1747270801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00f58a0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f71d4009fb0 tx=0x7f71d4009b30).stop 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.865+0000 7f71e4b89700 1 -- 172.21.15.76:0/1747270801 shutdown_connections 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.865+0000 7f71e4b89700 1 --2- 172.21.15.76:0/1747270801 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00f58a0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.865+0000 7f71e4b89700 1 -- 172.21.15.76:0/1747270801 >> 172.21.15.76:0/1747270801 conn(0x7f71e00f0be0 msgr2=0x7f71e00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71e4b89700 1 -- 172.21.15.76:0/1747270801 shutdown_connections 2023-12-26T22:02:27.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71e4b89700 1 -- 172.21.15.76:0/1747270801 wait complete. 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71e4b89700 1 Processor -- start 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71e4b89700 1 -- start start 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71e4b89700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71e4b89700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71e00fb8e0 con 0x7f71e00f5480 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71de59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71de59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34864/0 (socket says 172.21.15.76:34864) 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.866+0000 7f71de59c700 1 -- 172.21.15.76:0/4145389531 learned_addr learned my addr 172.21.15.76:0/4145389531 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71de59c700 1 -- 172.21.15.76:0/4145389531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d4005040 con 0x7f71e00f5480 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71de59c700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00fd460 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f71d40089f0 tx=0x7f71d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71d4009320 con 0x7f71e00f5480 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71e00fba80 con 0x7f71e00f5480 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f71d4013ec0 con 0x7f71e00f5480 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f71d4014d30 con 0x7f71e00f5480 2023-12-26T22:02:27.868 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.867+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71e00fbf10 con 0x7f71e00f5480 2023-12-26T22:02:27.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.868+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f71d4014420 con 0x7f71e00f5480 2023-12-26T22:02:27.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.868+0000 7f71ceffd700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71c8031090 0x7f71c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:27.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.868+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71e00591e0 con 0x7f71e00f5480 2023-12-26T22:02:27.869 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.868+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f71d4042400 con 0x7f71e00f5480 2023-12-26T22:02:27.870 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.868+0000 7f71ddd9b700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71c8031090 0x7f71c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:27.870 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.869+0000 7f71ddd9b700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71c8031090 0x7f71c8033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f71d00096a0 tx=0x7f71d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:27.873 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:27.873+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f71d401d980 con 0x7f71e00f5480 2023-12-26T22:02:28.084 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.084+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f71e00fc6b0 con 0x7f71e00f5480 2023-12-26T22:02:28.084 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.084+0000 7f71ceffd700 1 -- 172.21.15.76:0/4145389531 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f71d401d380 con 0x7f71e00f5480 2023-12-26T22:02:28.085 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:28.085 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:28.086 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71c8031090 msgr2=0x7f71c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:28.086 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71c8031090 0x7f71c8033550 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f71d00096a0 tx=0x7f71d0006b40).stop 2023-12-26T22:02:28.086 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 msgr2=0x7f71e00fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:28.086 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00fd460 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f71d40089f0 tx=0x7f71d400b040).stop 2023-12-26T22:02:28.086 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 shutdown_connections 2023-12-26T22:02:28.086 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f71c8031090 0x7f71c8033550 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:28.087 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 --2- 172.21.15.76:0/4145389531 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f71e00f5480 0x7f71e00fd460 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:28.087 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 >> 172.21.15.76:0/4145389531 conn(0x7f71e00f0be0 msgr2=0x7f71e005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:28.087 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 shutdown_connections 2023-12-26T22:02:28.087 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:28.086+0000 7f71e4b89700 1 -- 172.21.15.76:0/4145389531 wait complete. 2023-12-26T22:02:28.087 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:27.378985+0000 mgr.smithi059.gwkiyc (mgr.14156) 179 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:28.086043+0000 mon.smithi059 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.76:0/4145389531' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:29.467 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:29.467 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:30.692 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 -- 172.21.15.76:0/3931315386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 msgr2=0x7fca0c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:30.692 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 --2- 172.21.15.76:0/3931315386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c0f58a0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fca04009960 tx=0x7fca04004d10).stop 2023-12-26T22:02:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 -- 172.21.15.76:0/3931315386 shutdown_connections 2023-12-26T22:02:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 --2- 172.21.15.76:0/3931315386 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c0f58a0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 -- 172.21.15.76:0/3931315386 >> 172.21.15.76:0/3931315386 conn(0x7fca0c0f0be0 msgr2=0x7fca0c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 -- 172.21.15.76:0/3931315386 shutdown_connections 2023-12-26T22:02:30.693 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.692+0000 7fca12b91700 1 -- 172.21.15.76:0/3931315386 wait complete. 2023-12-26T22:02:30.694 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca12b91700 1 Processor -- start 2023-12-26T22:02:30.694 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca12b91700 1 -- start start 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca12b91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca12b91700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca0c060320 con 0x7fca0c0f5480 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca1092d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca1092d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34882/0 (socket says 172.21.15.76:34882) 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.693+0000 7fca1092d700 1 -- 172.21.15.76:0/3443848920 learned_addr learned my addr 172.21.15.76:0/3443848920 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca1092d700 1 -- 172.21.15.76:0/3443848920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca04009640 con 0x7fca0c0f5480 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca1092d700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c1094f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fca04009610 tx=0x7fca04004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca04018070 con 0x7fca0c0f5480 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca0c060520 con 0x7fca0c0f5480 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fca0400b7e0 con 0x7fca0c0f5480 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca0401d420 con 0x7fca0c0f5480 2023-12-26T22:02:30.695 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.694+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca0c109ba0 con 0x7fca0c0f5480 2023-12-26T22:02:30.696 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.695+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fca04024060 con 0x7fca0c0f5480 2023-12-26T22:02:30.696 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.695+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca0c045bb0 con 0x7fca0c0f5480 2023-12-26T22:02:30.696 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.695+0000 7fca097fa700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc9f4031090 0x7fc9f4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:30.696 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.695+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fca04042570 con 0x7fca0c0f5480 2023-12-26T22:02:30.696 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.695+0000 7fca0bfff700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc9f4031090 0x7fc9f4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:30.697 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.696+0000 7fca0bfff700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc9f4031090 0x7fc9f4033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fca000096a0 tx=0x7fca00006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:30.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.700+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fca040116a0 con 0x7fca0c0f5480 2023-12-26T22:02:30.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.905+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca0c0faf70 con 0x7fca0c0f5480 2023-12-26T22:02:30.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.906+0000 7fca097fa700 1 -- 172.21.15.76:0/3443848920 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca040118b0 con 0x7fca0c0f5480 2023-12-26T22:02:30.906 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:30.906 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:30.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc9f4031090 msgr2=0x7fc9f4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:30.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc9f4031090 0x7fc9f4033550 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fca000096a0 tx=0x7fca00006b40).stop 2023-12-26T22:02:30.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 msgr2=0x7fca0c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:30.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c1094f0 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fca04009610 tx=0x7fca04004d10).stop 2023-12-26T22:02:30.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 shutdown_connections 2023-12-26T22:02:30.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fc9f4031090 0x7fc9f4033550 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:30.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 --2- 172.21.15.76:0/3443848920 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca0c0f5480 0x7fca0c1094f0 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:30.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 >> 172.21.15.76:0/3443848920 conn(0x7fca0c0f0be0 msgr2=0x7fca0c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:30.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 shutdown_connections 2023-12-26T22:02:30.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:30.907+0000 7fca12b91700 1 -- 172.21.15.76:0/3443848920 wait complete. 2023-12-26T22:02:30.908 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:31.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:29.379416+0000 mgr.smithi059.gwkiyc (mgr.14156) 180 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:32.287 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:32.288 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:32.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:30.907131+0000 mon.smithi059 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.76:0/3443848920' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:31.379810+0000 mgr.smithi059.gwkiyc (mgr.14156) 181 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:33.454 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- 172.21.15.76:0/1575343640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 msgr2=0x7fa3040f67f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 --2- 172.21.15.76:0/1575343640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa3040f67f0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa2f4009fb0 tx=0x7fa2f4009b30).stop 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- 172.21.15.76:0/1575343640 shutdown_connections 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 --2- 172.21.15.76:0/1575343640 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa3040f67f0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- 172.21.15.76:0/1575343640 >> 172.21.15.76:0/1575343640 conn(0x7fa3040f1b10 msgr2=0x7fa3040f3f50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- 172.21.15.76:0/1575343640 shutdown_connections 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- 172.21.15.76:0/1575343640 wait complete. 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 Processor -- start 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- start start 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa30406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.454+0000 7fa30ab6a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa304069f00 con 0x7fa3040f63d0 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa309b68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa30406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa309b68700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa30406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34894/0 (socket says 172.21.15.76:34894) 2023-12-26T22:02:33.455 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa309b68700 1 -- 172.21.15.76:0/817888860 learned_addr learned my addr 172.21.15.76:0/817888860 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa309b68700 1 -- 172.21.15.76:0/817888860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f4005040 con 0x7fa3040f63d0 2023-12-26T22:02:33.457 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa309b68700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa30406ba80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fa2f40089f0 tx=0x7fa2f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2f4009320 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30406a0a0 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fa2f4013ec0 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa2f4014d30 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.455+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30406a530 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.456+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fa2f40095f0 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.456+0000 7fa2fa7fc700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2f0031090 0x7fa2f0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.456+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fa2f40423e0 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.456+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3040591e0 con 0x7fa3040f63d0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.456+0000 7fa309367700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2f0031090 0x7fa2f0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:33.458 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.457+0000 7fa309367700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2f0031090 0x7fa2f0033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa3000096a0 tx=0x7fa300006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:33.461 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.461+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fa2f401d7f0 con 0x7fa3040f63d0 2023-12-26T22:02:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.669+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa304045bb0 con 0x7fa3040f63d0 2023-12-26T22:02:33.669 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.669+0000 7fa2fa7fc700 1 -- 172.21.15.76:0/817888860 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fa2f401d7f0 con 0x7fa3040f63d0 2023-12-26T22:02:33.669 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:33.670 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2f0031090 msgr2=0x7fa2f0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2f0031090 0x7fa2f0033550 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fa3000096a0 tx=0x7fa300006b40).stop 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 msgr2=0x7fa30406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa30406ba80 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fa2f40089f0 tx=0x7fa2f400b040).stop 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 shutdown_connections 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fa2f0031090 0x7fa2f0033550 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 --2- 172.21.15.76:0/817888860 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fa3040f63d0 0x7fa30406ba80 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 >> 172.21.15.76:0/817888860 conn(0x7fa3040f1b10 msgr2=0x7fa3040f2650 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:33.671 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 shutdown_connections 2023-12-26T22:02:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:33.671+0000 7fa30ab6a700 1 -- 172.21.15.76:0/817888860 wait complete. 2023-12-26T22:02:33.672 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:34.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:33 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:33.670917+0000 mon.smithi059 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.76:0/817888860' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:35.060 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:35.060 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:33.380220+0000 mgr.smithi059.gwkiyc (mgr.14156) 182 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.270+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2420677485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 msgr2=0x7f4a100f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.270+0000 7f4a16fb5700 1 --2- 172.21.15.76:0/2420677485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a100f58a0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f4a04009fb0 tx=0x7f4a04009b30).stop 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.270+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2420677485 shutdown_connections 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.271+0000 7f4a16fb5700 1 --2- 172.21.15.76:0/2420677485 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a100f58a0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.271+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2420677485 >> 172.21.15.76:0/2420677485 conn(0x7f4a100f0be0 msgr2=0x7f4a100f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.271+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2420677485 shutdown_connections 2023-12-26T22:02:36.271 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.271+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2420677485 wait complete. 2023-12-26T22:02:36.272 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.271+0000 7f4a16fb5700 1 Processor -- start 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.271+0000 7f4a16fb5700 1 -- start start 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a16fb5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a101056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a16fb5700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a10105be0 con 0x7f4a100f5480 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a14d51700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a101056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a14d51700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a101056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46314/0 (socket says 172.21.15.76:46314) 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a14d51700 1 -- 172.21.15.76:0/2000510269 learned_addr learned my addr 172.21.15.76:0/2000510269 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a14d51700 1 -- 172.21.15.76:0/2000510269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a04005040 con 0x7f4a100f5480 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.272+0000 7f4a14d51700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a101056f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f4a040089f0 tx=0x7f4a0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.273+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a04009320 con 0x7f4a100f5480 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.273+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4a04013ec0 con 0x7f4a100f5480 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.273+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a10105de0 con 0x7f4a100f5480 2023-12-26T22:02:36.273 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.273+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a04014d30 con 0x7f4a100f5480 2023-12-26T22:02:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.273+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a10106270 con 0x7f4a100f5480 2023-12-26T22:02:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.274+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4a040263a0 con 0x7f4a100f5480 2023-12-26T22:02:36.274 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.274+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a10045bb0 con 0x7f4a100f5480 2023-12-26T22:02:36.275 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.274+0000 7f4a0d7fa700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4a00031090 0x7f4a00033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:36.275 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.274+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f4a04042500 con 0x7f4a100f5480 2023-12-26T22:02:36.275 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.274+0000 7f4a0ffff700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4a00031090 0x7f4a00033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:36.275 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.275+0000 7f4a0ffff700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4a00031090 0x7f4a00033550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f49fc0096a0 tx=0x7f49fc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:36.279 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.279+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f4a04012920 con 0x7f4a100f5480 2023-12-26T22:02:36.485 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.486+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a100faf70 con 0x7f4a100f5480 2023-12-26T22:02:36.486 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.486+0000 7f4a0d7fa700 1 -- 172.21.15.76:0/2000510269 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f4a04012330 con 0x7f4a100f5480 2023-12-26T22:02:36.487 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:36.487 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4a00031090 msgr2=0x7f4a00033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4a00031090 0x7f4a00033550 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f49fc0096a0 tx=0x7f49fc006b40).stop 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 msgr2=0x7f4a101056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a101056f0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f4a040089f0 tx=0x7f4a0400b040).stop 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 shutdown_connections 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4a00031090 0x7f4a00033550 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 --2- 172.21.15.76:0/2000510269 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f4a100f5480 0x7f4a101056f0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:36.488 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 >> 172.21.15.76:0/2000510269 conn(0x7f4a100f0be0 msgr2=0x7f4a100f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:36.489 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 shutdown_connections 2023-12-26T22:02:36.489 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:36.488+0000 7f4a16fb5700 1 -- 172.21.15.76:0/2000510269 wait complete. 2023-12-26T22:02:36.489 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:35.380627+0000 mgr.smithi059.gwkiyc (mgr.14156) 183 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:36.487985+0000 mon.smithi059 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.76:0/2000510269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:37.869 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:37.869 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:39.048 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 -- 172.21.15.76:0/1008827032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 msgr2=0x7fdc6c0f8fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 --2- 172.21.15.76:0/1008827032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c0f8fb0 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fdc5c009fb0 tx=0x7fdc5c009b30).stop 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 -- 172.21.15.76:0/1008827032 shutdown_connections 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 --2- 172.21.15.76:0/1008827032 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c0f8fb0 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 -- 172.21.15.76:0/1008827032 >> 172.21.15.76:0/1008827032 conn(0x7fdc6c0f0a30 msgr2=0x7fdc6c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 -- 172.21.15.76:0/1008827032 shutdown_connections 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 -- 172.21.15.76:0/1008827032 wait complete. 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 Processor -- start 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.048+0000 7fdc70eec700 1 -- start start 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc70eec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c1065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc70eec700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc6c106aa0 con 0x7fdc6c060360 2023-12-26T22:02:39.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c1065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:39.050 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc6b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c1065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46338/0 (socket says 172.21.15.76:46338) 2023-12-26T22:02:39.050 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc6b7fe700 1 -- 172.21.15.76:0/1436139606 learned_addr learned my addr 172.21.15.76:0/1436139606 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc6b7fe700 1 -- 172.21.15.76:0/1436139606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc5c005040 con 0x7fdc6c060360 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc6b7fe700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c1065b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fdc5c009f80 tx=0x7fdc5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5c009320 con 0x7fdc6c060360 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc6c106ca0 con 0x7fdc6c060360 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdc5c013680 con 0x7fdc6c060360 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc5c01eda0 con 0x7fdc6c060360 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.049+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc6c107130 con 0x7fdc6c060360 2023-12-26T22:02:39.051 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.050+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdc5c01e420 con 0x7fdc6c060360 2023-12-26T22:02:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.050+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc6c045bb0 con 0x7fdc6c060360 2023-12-26T22:02:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.050+0000 7fdc63fff700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc580310e0 0x7fdc580335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.050+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdc5c042440 con 0x7fdc6c060360 2023-12-26T22:02:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.051+0000 7fdc6affd700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc580310e0 0x7fdc580335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:39.052 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.051+0000 7fdc6affd700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc580310e0 0x7fdc580335a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fdc540096a0 tx=0x7fdc54006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:39.055 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.055+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdc5c01d380 con 0x7fdc6c060360 2023-12-26T22:02:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.262+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc6c107fe0 con 0x7fdc6c060360 2023-12-26T22:02:39.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.262+0000 7fdc63fff700 1 -- 172.21.15.76:0/1436139606 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdc5c004490 con 0x7fdc6c060360 2023-12-26T22:02:39.262 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:39.263 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc580310e0 msgr2=0x7fdc580335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc580310e0 0x7fdc580335a0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fdc540096a0 tx=0x7fdc54006b40).stop 2023-12-26T22:02:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 msgr2=0x7fdc6c1065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c1065b0 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fdc5c009f80 tx=0x7fdc5c00b040).stop 2023-12-26T22:02:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 shutdown_connections 2023-12-26T22:02:39.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc580310e0 0x7fdc580335a0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 --2- 172.21.15.76:0/1436139606 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdc6c060360 0x7fdc6c1065b0 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 >> 172.21.15.76:0/1436139606 conn(0x7fdc6c0f0a30 msgr2=0x7fdc6c0f14e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 shutdown_connections 2023-12-26T22:02:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:39.264+0000 7fdc70eec700 1 -- 172.21.15.76:0/1436139606 wait complete. 2023-12-26T22:02:39.265 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:39.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:37.380958+0000 mgr.smithi059.gwkiyc (mgr.14156) 184 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:40.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:39 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:39.264126+0000 mon.smithi059 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.76:0/1436139606' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:40.655 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:40.656 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:39.381285+0000 mgr.smithi059.gwkiyc (mgr.14156) 185 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 -- 172.21.15.76:0/1161159053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 msgr2=0x7f710c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 --2- 172.21.15.76:0/1161159053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c0f58a0 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f70fc009fb0 tx=0x7f70fc009b30).stop 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 -- 172.21.15.76:0/1161159053 shutdown_connections 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 --2- 172.21.15.76:0/1161159053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c0f58a0 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 -- 172.21.15.76:0/1161159053 >> 172.21.15.76:0/1161159053 conn(0x7f710c0f0be0 msgr2=0x7f710c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 -- 172.21.15.76:0/1161159053 shutdown_connections 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.847+0000 7f7112934700 1 -- 172.21.15.76:0/1161159053 wait complete. 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.848+0000 7f7112934700 1 Processor -- start 2023-12-26T22:02:41.848 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.848+0000 7f7112934700 1 -- start start 2023-12-26T22:02:41.850 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.848+0000 7f7112934700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c1056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:41.850 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.848+0000 7f7112934700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f710c105be0 con 0x7f710c0f5480 2023-12-26T22:02:41.850 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.848+0000 7f710bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c1056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:41.850 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f710bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c1056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46358/0 (socket says 172.21.15.76:46358) 2023-12-26T22:02:41.850 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f710bfff700 1 -- 172.21.15.76:0/3925842208 learned_addr learned my addr 172.21.15.76:0/3925842208 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:41.850 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f710bfff700 1 -- 172.21.15.76:0/3925842208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70fc005040 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f710bfff700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c1056f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f70fc0089f0 tx=0x7f70fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70fc009320 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f710c105de0 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f70fc013ec0 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f70fc014d30 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.849+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f710c106270 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.850+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f70fc0095f0 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.850+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f710c045bb0 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.850+0000 7f7108ff9700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f70f40310e0 0x7f70f40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.850+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f70fc0424a0 con 0x7f710c0f5480 2023-12-26T22:02:41.851 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.850+0000 7f710b7fe700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f70f40310e0 0x7f70f40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:41.852 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.851+0000 7f710b7fe700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f70f40310e0 0x7f70f40335a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f71000096a0 tx=0x7f7100006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:41.855 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:41.855+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f70fc01d980 con 0x7f710c0f5480 2023-12-26T22:02:42.064 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.064+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f710c0faf70 con 0x7f710c0f5480 2023-12-26T22:02:42.065 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.066+0000 7f7108ff9700 1 -- 172.21.15.76:0/3925842208 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f70fc01d380 con 0x7f710c0f5480 2023-12-26T22:02:42.065 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:42.065 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f70f40310e0 msgr2=0x7f70f40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f70f40310e0 0x7f70f40335a0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f71000096a0 tx=0x7f7100006b40).stop 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 msgr2=0x7f710c1056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c1056f0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f70fc0089f0 tx=0x7f70fc00b040).stop 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 shutdown_connections 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f70f40310e0 0x7f70f40335a0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:42.067 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 --2- 172.21.15.76:0/3925842208 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f710c0f5480 0x7f710c1056f0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:42.068 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 >> 172.21.15.76:0/3925842208 conn(0x7f710c0f0be0 msgr2=0x7f710c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:42.068 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 shutdown_connections 2023-12-26T22:02:42.068 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:42.067+0000 7f7112934700 1 -- 172.21.15.76:0/3925842208 wait complete. 2023-12-26T22:02:42.068 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:41.381609+0000 mgr.smithi059.gwkiyc (mgr.14156) 186 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:42.066596+0000 mon.smithi059 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.76:0/3925842208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:43.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:43.455 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.714+0000 7f6072427700 1 -- 172.21.15.76:0/3505520871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 msgr2=0x7f606c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.714+0000 7f6072427700 1 --2- 172.21.15.76:0/3505520871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c0f58a0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f605c009fb0 tx=0x7f605c009b30).stop 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.714+0000 7f6072427700 1 -- 172.21.15.76:0/3505520871 shutdown_connections 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.714+0000 7f6072427700 1 --2- 172.21.15.76:0/3505520871 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c0f58a0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.714+0000 7f6072427700 1 -- 172.21.15.76:0/3505520871 >> 172.21.15.76:0/3505520871 conn(0x7f606c0f0be0 msgr2=0x7f606c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.715+0000 7f6072427700 1 -- 172.21.15.76:0/3505520871 shutdown_connections 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.715+0000 7f6072427700 1 -- 172.21.15.76:0/3505520871 wait complete. 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.715+0000 7f6072427700 1 Processor -- start 2023-12-26T22:02:44.714 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.715+0000 7f6072427700 1 -- start start 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.715+0000 7f6072427700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.715+0000 7f6072427700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f606c060320 con 0x7f606c0f5480 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f606bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f606bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46372/0 (socket says 172.21.15.76:46372) 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f606bfff700 1 -- 172.21.15.76:0/3516107810 learned_addr learned my addr 172.21.15.76:0/3516107810 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f606bfff700 1 -- 172.21.15.76:0/3516107810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f605c005040 con 0x7f606c0f5480 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f606bfff700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c1094f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f605c0089f0 tx=0x7f605c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f605c009320 con 0x7f606c0f5480 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f606c060520 con 0x7f606c0f5480 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f605c013ec0 con 0x7f606c0f5480 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f605c014d30 con 0x7f606c0f5480 2023-12-26T22:02:44.716 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.716+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f606c109ba0 con 0x7f606c0f5480 2023-12-26T22:02:44.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.717+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f605c0095f0 con 0x7f606c0f5480 2023-12-26T22:02:44.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.717+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f606c045bb0 con 0x7f606c0f5480 2023-12-26T22:02:44.717 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.717+0000 7f6068ff9700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6054031090 0x7f6054033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:44.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.717+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f605c042420 con 0x7f606c0f5480 2023-12-26T22:02:44.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.718+0000 7f606b7fe700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6054031090 0x7f6054033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:44.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.718+0000 7f606b7fe700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6054031090 0x7f6054033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f60600096a0 tx=0x7f6060006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:44.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.722+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f605c01d7f0 con 0x7f606c0f5480 2023-12-26T22:02:44.930 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.931+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f606c0faf70 con 0x7f606c0f5480 2023-12-26T22:02:44.931 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.932+0000 7f6068ff9700 1 -- 172.21.15.76:0/3516107810 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f605c00fe40 con 0x7f606c0f5480 2023-12-26T22:02:44.931 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:44.931 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6054031090 msgr2=0x7f6054033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6054031090 0x7f6054033550 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f60600096a0 tx=0x7f6060006b40).stop 2023-12-26T22:02:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 msgr2=0x7f606c1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:44.932 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c1094f0 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f605c0089f0 tx=0x7f605c00b040).stop 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 shutdown_connections 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f6054031090 0x7f6054033550 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 --2- 172.21.15.76:0/3516107810 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f606c0f5480 0x7f606c1094f0 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 >> 172.21.15.76:0/3516107810 conn(0x7f606c0f0be0 msgr2=0x7f606c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 shutdown_connections 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:44.933+0000 7f6072427700 1 -- 172.21.15.76:0/3516107810 wait complete. 2023-12-26T22:02:44.933 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:45.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:43.381990+0000 mgr.smithi059.gwkiyc (mgr.14156) 187 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:44.932449+0000 mon.smithi059 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.76:0/3516107810' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:45.470802+0000 mon.smithi059 (mon.0) 249 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T22:02:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:45.471007+0000 mon.smithi059 (mon.0) 250 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T22:02:46.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:46.317 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:47.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:45.382432+0000 mgr.smithi059.gwkiyc (mgr.14156) 188 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:47.464 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2341160695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c80f54b0 msgr2=0x7ff6c80f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:47.464 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 --2- 172.21.15.76:0/2341160695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c80f54b0 0x7ff6c80f58d0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7ff6bc009fb0 tx=0x7ff6bc009b30).stop 2023-12-26T22:02:47.464 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2341160695 shutdown_connections 2023-12-26T22:02:47.464 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 --2- 172.21.15.76:0/2341160695 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c80f54b0 0x7ff6c80f58d0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:47.464 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2341160695 >> 172.21.15.76:0/2341160695 conn(0x7ff6c80f0c10 msgr2=0x7ff6c80f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:47.464 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2341160695 shutdown_connections 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.461+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2341160695 wait complete. 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6cd88e700 1 Processor -- start 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6cd88e700 1 -- start start 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6cd88e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 0x7ff6c8104b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6cd88e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6c8104ff0 con 0x7ff6c81046e0 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 0x7ff6c8104b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6c6ffd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 0x7ff6c8104b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46100/0 (socket says 172.21.15.76:46100) 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.462+0000 7ff6c6ffd700 1 -- 172.21.15.76:0/2123876260 learned_addr learned my addr 172.21.15.76:0/2123876260 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6c6ffd700 1 -- 172.21.15.76:0/2123876260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc005040 con 0x7ff6c81046e0 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6c6ffd700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 0x7ff6c8104b00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ff6bc009880 tx=0x7ff6bc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6bc013680 con 0x7ff6c81046e0 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff6bc0095f0 con 0x7ff6c81046e0 2023-12-26T22:02:47.465 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6c81051f0 con 0x7ff6c81046e0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6bc014e40 con 0x7ff6c81046e0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.463+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6c8105d40 con 0x7ff6c81046e0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.464+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff6bc014420 con 0x7ff6c81046e0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.464+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6c80591e0 con 0x7ff6c81046e0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.464+0000 7ff6af7fe700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff6b00332e0 0x7ff6b00357a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.464+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff6bc042410 con 0x7ff6c81046e0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.464+0000 7ff6c67fc700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff6b00332e0 0x7ff6b00357a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:47.466 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.464+0000 7ff6c67fc700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff6b00332e0 0x7ff6b00357a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff6b80096a0 tx=0x7ff6b8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:47.468 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.469+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff6bc025cc0 con 0x7ff6c81046e0 2023-12-26T22:02:47.678 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.680+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6c8106500 con 0x7ff6c81046e0 2023-12-26T22:02:47.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.680+0000 7ff6af7fe700 1 -- 172.21.15.76:0/2123876260 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff6bc025ae0 con 0x7ff6c81046e0 2023-12-26T22:02:47.679 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:47.679 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff6b00332e0 msgr2=0x7ff6b00357a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff6b00332e0 0x7ff6b00357a0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ff6b80096a0 tx=0x7ff6b8006b40).stop 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 msgr2=0x7ff6c8104b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 0x7ff6c8104b00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7ff6bc009880 tx=0x7ff6bc009b30).stop 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 shutdown_connections 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff6b00332e0 0x7ff6b00357a0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:47.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 --2- 172.21.15.76:0/2123876260 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff6c81046e0 0x7ff6c8104b00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:47.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 >> 172.21.15.76:0/2123876260 conn(0x7ff6c80f0c10 msgr2=0x7ff6c80f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:47.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 shutdown_connections 2023-12-26T22:02:47.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:47.682+0000 7ff6cd88e700 1 -- 172.21.15.76:0/2123876260 wait complete. 2023-12-26T22:02:47.682 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:48.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:47 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:47.680887+0000 mon.smithi059 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.76:0/2123876260' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:49.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:49.062 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:47.382775+0000 mgr.smithi059.gwkiyc (mgr.14156) 189 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:50.323 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.324+0000 7f974e0d1700 1 -- 172.21.15.76:0/3791855409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 msgr2=0x7f97480f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.324+0000 7f974e0d1700 1 --2- 172.21.15.76:0/3791855409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f97480f9160 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f9738009960 tx=0x7f9738004d10).stop 2023-12-26T22:02:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.324+0000 7f974e0d1700 1 -- 172.21.15.76:0/3791855409 shutdown_connections 2023-12-26T22:02:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.324+0000 7f974e0d1700 1 --2- 172.21.15.76:0/3791855409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f97480f9160 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.324+0000 7f974e0d1700 1 -- 172.21.15.76:0/3791855409 >> 172.21.15.76:0/3791855409 conn(0x7f97480f0be0 msgr2=0x7f97480f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.325+0000 7f974e0d1700 1 -- 172.21.15.76:0/3791855409 shutdown_connections 2023-12-26T22:02:50.324 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.325+0000 7f974e0d1700 1 -- 172.21.15.76:0/3791855409 wait complete. 2023-12-26T22:02:50.325 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.325+0000 7f974e0d1700 1 Processor -- start 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.325+0000 7f974e0d1700 1 -- start start 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.325+0000 7f974e0d1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f9748108970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.325+0000 7f974e0d1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9748108e60 con 0x7f9748060360 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f97477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f9748108970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f97477fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f9748108970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46116/0 (socket says 172.21.15.76:46116) 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f97477fe700 1 -- 172.21.15.76:0/25776699 learned_addr learned my addr 172.21.15.76:0/25776699 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f97477fe700 1 -- 172.21.15.76:0/25776699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9738009640 con 0x7f9748060360 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f97477fe700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f9748108970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9738009610 tx=0x7f9738004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9738018070 con 0x7f9748060360 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9748109060 con 0x7f9748060360 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f973800b7e0 con 0x7f9748060360 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f973801d420 con 0x7f9748060360 2023-12-26T22:02:50.326 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.326+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97481094f0 con 0x7f9748060360 2023-12-26T22:02:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.327+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9738024060 con 0x7f9748060360 2023-12-26T22:02:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.327+0000 7f973ffff700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9734031090 0x7f9734033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.327+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9748045bb0 con 0x7f9748060360 2023-12-26T22:02:50.327 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.327+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9738042610 con 0x7f9748060360 2023-12-26T22:02:50.328 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.328+0000 7f9746ffd700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9734031090 0x7f9734033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:50.328 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.328+0000 7f9746ffd700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9734031090 0x7f9734033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f97300096a0 tx=0x7f9730006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:50.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.332+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9738011b60 con 0x7f9748060360 2023-12-26T22:02:50.537 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.538+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f974810a2f0 con 0x7f9748060360 2023-12-26T22:02:50.538 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.539+0000 7f973ffff700 1 -- 172.21.15.76:0/25776699 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9738011930 con 0x7f9748060360 2023-12-26T22:02:50.538 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:50.538 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:50.539 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9734031090 msgr2=0x7f9734033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9734031090 0x7f9734033550 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f97300096a0 tx=0x7f9730006b40).stop 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 msgr2=0x7f9748108970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f9748108970 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f9738009610 tx=0x7f9738004d10).stop 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 shutdown_connections 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9734031090 0x7f9734033550 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 --2- 172.21.15.76:0/25776699 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9748060360 0x7f9748108970 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.540+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 >> 172.21.15.76:0/25776699 conn(0x7f97480f0be0 msgr2=0x7f97480f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.541+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 shutdown_connections 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:50.541+0000 7f974e0d1700 1 -- 172.21.15.76:0/25776699 wait complete. 2023-12-26T22:02:50.540 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:49.383202+0000 mgr.smithi059.gwkiyc (mgr.14156) 190 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:50.539586+0000 mon.smithi059 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.76:0/25776699' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:51.908 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:51.908 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 -- 172.21.15.76:0/920070271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 msgr2=0x7f67c40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 --2- 172.21.15.76:0/920070271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c40f58a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f67c0009fb0 tx=0x7f67c0009b30).stop 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 -- 172.21.15.76:0/920070271 shutdown_connections 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 --2- 172.21.15.76:0/920070271 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c40f58a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 -- 172.21.15.76:0/920070271 >> 172.21.15.76:0/920070271 conn(0x7f67c40f0be0 msgr2=0x7f67c40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 -- 172.21.15.76:0/920070271 shutdown_connections 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 -- 172.21.15.76:0/920070271 wait complete. 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 Processor -- start 2023-12-26T22:02:53.089 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 -- start start 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.089+0000 7f67cb5ee700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c41067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67cb5ee700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c4106cd0 con 0x7f67c40f5480 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67c938a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c41067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67c938a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c41067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:46140/0 (socket says 172.21.15.76:46140) 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67c938a700 1 -- 172.21.15.76:0/2248403685 learned_addr learned my addr 172.21.15.76:0/2248403685 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67c938a700 1 -- 172.21.15.76:0/2248403685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67c0005040 con 0x7f67c40f5480 2023-12-26T22:02:53.090 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67c938a700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c41067e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f67c00089f0 tx=0x7f67c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67c0009320 con 0x7f67c40f5480 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c4106ed0 con 0x7f67c40f5480 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f67c0013ec0 con 0x7f67c40f5480 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.090+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67c0014d30 con 0x7f67c40f5480 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.091+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c4107360 con 0x7f67c40f5480 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.091+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f67c00095f0 con 0x7f67c40f5480 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.091+0000 7f67b9ffb700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f67b0031090 0x7f67b0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:53.091 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.091+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c4045bb0 con 0x7f67c40f5480 2023-12-26T22:02:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.092+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f67c00423e0 con 0x7f67c40f5480 2023-12-26T22:02:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.092+0000 7f67c8b89700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f67b0031090 0x7f67b0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:53.092 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.092+0000 7f67c8b89700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f67b0031090 0x7f67b0033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f67b40096a0 tx=0x7f67b4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:53.097 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.097+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f67c000fe40 con 0x7f67c40f5480 2023-12-26T22:02:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:51.383583+0000 mgr.smithi059.gwkiyc (mgr.14156) 191 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:53.303 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.304+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67c4108210 con 0x7f67c40f5480 2023-12-26T22:02:53.304 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.305+0000 7f67b9ffb700 1 -- 172.21.15.76:0/2248403685 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f67c000fe40 con 0x7f67c40f5480 2023-12-26T22:02:53.304 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:53.304 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:53.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f67b0031090 msgr2=0x7f67b0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:53.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f67b0031090 0x7f67b0033550 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f67b40096a0 tx=0x7f67b4006b40).stop 2023-12-26T22:02:53.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 msgr2=0x7f67c41067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:53.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c41067e0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f67c00089f0 tx=0x7f67c000b040).stop 2023-12-26T22:02:53.306 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 shutdown_connections 2023-12-26T22:02:53.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f67b0031090 0x7f67b0033550 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:53.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 --2- 172.21.15.76:0/2248403685 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f67c40f5480 0x7f67c41067e0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:53.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 >> 172.21.15.76:0/2248403685 conn(0x7f67c40f0be0 msgr2=0x7f67c405ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:53.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 shutdown_connections 2023-12-26T22:02:53.307 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:53.306+0000 7f67cb5ee700 1 -- 172.21.15.76:0/2248403685 wait complete. 2023-12-26T22:02:53.307 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:54.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:53 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:53.305287+0000 mon.smithi059 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.76:0/2248403685' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:54.687 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:54.688 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:55.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:53.383863+0000 mgr.smithi059.gwkiyc (mgr.14156) 192 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:55.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 -- 172.21.15.76:0/49595774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 msgr2=0x7f25880f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:55.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 --2- 172.21.15.76:0/49595774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25880f58a0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f257c009fb0 tx=0x7f257c009b30).stop 2023-12-26T22:02:55.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 -- 172.21.15.76:0/49595774 shutdown_connections 2023-12-26T22:02:55.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 --2- 172.21.15.76:0/49595774 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25880f58a0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:55.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 -- 172.21.15.76:0/49595774 >> 172.21.15.76:0/49595774 conn(0x7f25880f0be0 msgr2=0x7f25880f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:55.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 -- 172.21.15.76:0/49595774 shutdown_connections 2023-12-26T22:02:55.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.830+0000 7f258c8cd700 1 -- 172.21.15.76:0/49595774 wait complete. 2023-12-26T22:02:55.831 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258c8cd700 1 Processor -- start 2023-12-26T22:02:55.831 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258c8cd700 1 -- start start 2023-12-26T22:02:55.831 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258c8cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25881056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:55.831 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258c8cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2588105be0 con 0x7f25880f5480 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25881056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25881056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34632/0 (socket says 172.21.15.76:34632) 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.831+0000 7f258659c700 1 -- 172.21.15.76:0/79836433 learned_addr learned my addr 172.21.15.76:0/79836433 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f258659c700 1 -- 172.21.15.76:0/79836433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f257c005040 con 0x7f25880f5480 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f258659c700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25881056f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f257c0089f0 tx=0x7f257c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257c009320 con 0x7f25880f5480 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2588105de0 con 0x7f25880f5480 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f257c013ec0 con 0x7f25880f5480 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f257c014d30 con 0x7f25880f5480 2023-12-26T22:02:55.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.832+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2588106270 con 0x7f25880f5480 2023-12-26T22:02:55.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.833+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f257c0095f0 con 0x7f25880f5480 2023-12-26T22:02:55.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.833+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2588045bb0 con 0x7f25880f5480 2023-12-26T22:02:55.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.833+0000 7f2576ffd700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2570031090 0x7f2570033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:55.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.833+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f257c0424a0 con 0x7f25880f5480 2023-12-26T22:02:55.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.833+0000 7f2585d9b700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2570031090 0x7f2570033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:55.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.834+0000 7f2585d9b700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2570031090 0x7f2570033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f25780096a0 tx=0x7f2578006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:55.836 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:55.838+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f257c01d980 con 0x7f25880f5480 2023-12-26T22:02:56.046 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.047+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25880faf70 con 0x7f25880f5480 2023-12-26T22:02:56.047 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.048+0000 7f2576ffd700 1 -- 172.21.15.76:0/79836433 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f257c01d380 con 0x7f25880f5480 2023-12-26T22:02:56.047 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:56.047 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2570031090 msgr2=0x7f2570033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:56.048 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2570031090 0x7f2570033550 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f25780096a0 tx=0x7f2578006b40).stop 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 msgr2=0x7f25881056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25881056f0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f257c0089f0 tx=0x7f257c00b040).stop 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 shutdown_connections 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2570031090 0x7f2570033550 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 --2- 172.21.15.76:0/79836433 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f25880f5480 0x7f25881056f0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 >> 172.21.15.76:0/79836433 conn(0x7f25880f0be0 msgr2=0x7f25880f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 shutdown_connections 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:56.049+0000 7f258c8cd700 1 -- 172.21.15.76:0/79836433 wait complete. 2023-12-26T22:02:56.049 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:55.384252+0000 mgr.smithi059.gwkiyc (mgr.14156) 193 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:02:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:56.048450+0000 mon.smithi059 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.76:0/79836433' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:02:57.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:02:57.441 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:02:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 -- 172.21.15.76:0/1641052438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 msgr2=0x7fea440f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 --2- 172.21.15.76:0/1641052438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea440f9160 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7fea34009fb0 tx=0x7fea34009b30).stop 2023-12-26T22:02:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 -- 172.21.15.76:0/1641052438 shutdown_connections 2023-12-26T22:02:58.651 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 --2- 172.21.15.76:0/1641052438 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea440f9160 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 -- 172.21.15.76:0/1641052438 >> 172.21.15.76:0/1641052438 conn(0x7fea440f0be0 msgr2=0x7fea440f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 -- 172.21.15.76:0/1641052438 shutdown_connections 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.650+0000 7fea4c4aa700 1 -- 172.21.15.76:0/1641052438 wait complete. 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4c4aa700 1 Processor -- start 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4c4aa700 1 -- start start 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4c4aa700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea44106750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4c4aa700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea44106c40 con 0x7fea44060360 2023-12-26T22:02:58.652 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4a246700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea44106750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4a246700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea44106750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34644/0 (socket says 172.21.15.76:34644) 2023-12-26T22:02:58.653 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.651+0000 7fea4a246700 1 -- 172.21.15.76:0/2388848889 learned_addr learned my addr 172.21.15.76:0/2388848889 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea4a246700 1 -- 172.21.15.76:0/2388848889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea34005040 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea4a246700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea44106750 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fea34009f80 tx=0x7fea3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea34009320 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea44106e40 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fea34013680 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea3401eda0 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.652+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea441072d0 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.653+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fea3401e420 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.653+0000 7fea3affd700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fea300310e0 0x7fea300335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.653+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fea34042330 con 0x7fea44060360 2023-12-26T22:02:58.654 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.653+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea440591e0 con 0x7fea44060360 2023-12-26T22:02:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.653+0000 7fea49a45700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fea300310e0 0x7fea300335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:02:58.655 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.654+0000 7fea49a45700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fea300310e0 0x7fea300335a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fea400096a0 tx=0x7fea40006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:02:58.658 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.659+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fea3401d380 con 0x7fea44060360 2023-12-26T22:02:58.864 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.864+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea44045bb0 con 0x7fea44060360 2023-12-26T22:02:58.865 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.865+0000 7fea3affd700 1 -- 172.21.15.76:0/2388848889 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fea3401cad0 con 0x7fea44060360 2023-12-26T22:02:58.865 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:02:58.865 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:02:58.866 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.866+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fea300310e0 msgr2=0x7fea300335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.866+0000 7fea4c4aa700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fea300310e0 0x7fea300335a0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7fea400096a0 tx=0x7fea40006b40).stop 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.866+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 msgr2=0x7fea44106750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.866+0000 7fea4c4aa700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea44106750 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fea34009f80 tx=0x7fea3400b040).stop 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.867+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 shutdown_connections 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.867+0000 7fea4c4aa700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fea300310e0 0x7fea300335a0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.867+0000 7fea4c4aa700 1 --2- 172.21.15.76:0/2388848889 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fea44060360 0x7fea44106750 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.867+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 >> 172.21.15.76:0/2388848889 conn(0x7fea440f0be0 msgr2=0x7fea440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.867+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 shutdown_connections 2023-12-26T22:02:58.867 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:02:58.867+0000 7fea4c4aa700 1 -- 172.21.15.76:0/2388848889 wait complete. 2023-12-26T22:02:58.868 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:02:59.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:57.384534+0000 mgr.smithi059.gwkiyc (mgr.14156) 194 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:00.244 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:00.244 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:00.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:02:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:02:58.866460+0000 mon.smithi059 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.76:0/2388848889' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:02:59.384750+0000 mgr.smithi059.gwkiyc (mgr.14156) 195 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:01.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 -- 172.21.15.76:0/134885223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 msgr2=0x7f57740f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:01.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 --2- 172.21.15.76:0/134885223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57740f58a0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f575c009fb0 tx=0x7f575c009b30).stop 2023-12-26T22:03:01.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 -- 172.21.15.76:0/134885223 shutdown_connections 2023-12-26T22:03:01.429 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 --2- 172.21.15.76:0/134885223 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57740f58a0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 -- 172.21.15.76:0/134885223 >> 172.21.15.76:0/134885223 conn(0x7f57740f0be0 msgr2=0x7f57740f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 -- 172.21.15.76:0/134885223 shutdown_connections 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.428+0000 7f5779fd1700 1 -- 172.21.15.76:0/134885223 wait complete. 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f5779fd1700 1 Processor -- start 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f5779fd1700 1 -- start start 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f5779fd1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57741067e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f5779fd1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5774106cd0 con 0x7f57740f5480 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f57737fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57741067e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f57737fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57741067e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34652/0 (socket says 172.21.15.76:34652) 2023-12-26T22:03:01.430 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.429+0000 7f57737fe700 1 -- 172.21.15.76:0/3974201914 learned_addr learned my addr 172.21.15.76:0/3974201914 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f57737fe700 1 -- 172.21.15.76:0/3974201914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f575c005040 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f57737fe700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57741067e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f575c0089f0 tx=0x7f575c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575c009320 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5774106ed0 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f575c013ec0 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f575c014d50 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.430+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5774107360 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.431+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f575c0095f0 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.431+0000 7f576bfff700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5760031090 0x7f5760033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.431+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5774045bb0 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.431+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f575c0423b0 con 0x7f57740f5480 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.431+0000 7f5772ffd700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5760031090 0x7f5760033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:01.432 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.432+0000 7f5772ffd700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5760031090 0x7f5760033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f57640096a0 tx=0x7f5764006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:01.435 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.436+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f575c01cb00 con 0x7f57740f5480 2023-12-26T22:03:01.644 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.644+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5774108210 con 0x7f57740f5480 2023-12-26T22:03:01.645 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.645+0000 7f576bfff700 1 -- 172.21.15.76:0/3974201914 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f575c02b2d0 con 0x7f57740f5480 2023-12-26T22:03:01.645 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:01.645 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5760031090 msgr2=0x7f5760033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5760031090 0x7f5760033550 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f57640096a0 tx=0x7f5764006b40).stop 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 msgr2=0x7f57741067e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57741067e0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f575c0089f0 tx=0x7f575c00b040).stop 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 shutdown_connections 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5760031090 0x7f5760033550 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 --2- 172.21.15.76:0/3974201914 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f57740f5480 0x7f57741067e0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 >> 172.21.15.76:0/3974201914 conn(0x7f57740f0be0 msgr2=0x7f577405ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 shutdown_connections 2023-12-26T22:03:01.647 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:01.647+0000 7f5779fd1700 1 -- 172.21.15.76:0/3974201914 wait complete. 2023-12-26T22:03:01.648 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:02.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:01 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:01.646266+0000 mon.smithi059 (mon.0) 256 : audit [DBG] from='client.? 172.21.15.76:0/3974201914' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:03.047 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:03.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:01.385029+0000 mgr.smithi059.gwkiyc (mgr.14156) 196 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:04.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.228+0000 7efe34a0a700 1 -- 172.21.15.76:0/3373623336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 msgr2=0x7efe300f4b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:04.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.228+0000 7efe34a0a700 1 --2- 172.21.15.76:0/3373623336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe300f4b70 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7efe18009fb0 tx=0x7efe18009b30).stop 2023-12-26T22:03:04.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 -- 172.21.15.76:0/3373623336 shutdown_connections 2023-12-26T22:03:04.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 --2- 172.21.15.76:0/3373623336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe300f4b70 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:04.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 -- 172.21.15.76:0/3373623336 >> 172.21.15.76:0/3373623336 conn(0x7efe300f01d0 msgr2=0x7efe300f25f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:04.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 -- 172.21.15.76:0/3373623336 shutdown_connections 2023-12-26T22:03:04.229 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 -- 172.21.15.76:0/3373623336 wait complete. 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 Processor -- start 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 -- start start 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe30106820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.229+0000 7efe34a0a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe30106d10 con 0x7efe300f4750 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe2ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe30106820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe2ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe30106820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:34680/0 (socket says 172.21.15.76:34680) 2023-12-26T22:03:04.231 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe2ed9d700 1 -- 172.21.15.76:0/1065467309 learned_addr learned my addr 172.21.15.76:0/1065467309 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe2ed9d700 1 -- 172.21.15.76:0/1065467309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe18005040 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe2ed9d700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe30106820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7efe18009f80 tx=0x7efe1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe18009320 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe30106f10 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe301073a0 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7efe18013680 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.230+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efe1801ed00 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.231+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7efe18014420 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.231+0000 7efe277fe700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe1c031090 0x7efe1c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.231+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7efe180095f0 con 0x7efe300f4750 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.231+0000 7efe2e59c700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe1c031090 0x7efe1c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:04.232 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.231+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe300591e0 con 0x7efe300f4750 2023-12-26T22:03:04.233 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.232+0000 7efe2e59c700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe1c031090 0x7efe1c033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efe200096a0 tx=0x7efe20006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:04.236 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.236+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7efe1801ee60 con 0x7efe300f4750 2023-12-26T22:03:04.445 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.445+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efe30045bb0 con 0x7efe300f4750 2023-12-26T22:03:04.445 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.445+0000 7efe277fe700 1 -- 172.21.15.76:0/1065467309 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7efe1801c910 con 0x7efe300f4750 2023-12-26T22:03:04.446 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:04.446 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe1c031090 msgr2=0x7efe1c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe1c031090 0x7efe1c033550 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7efe200096a0 tx=0x7efe20006b40).stop 2023-12-26T22:03:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 msgr2=0x7efe30106820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe30106820 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7efe18009f80 tx=0x7efe1800b040).stop 2023-12-26T22:03:04.447 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 shutdown_connections 2023-12-26T22:03:04.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7efe1c031090 0x7efe1c033550 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:04.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 --2- 172.21.15.76:0/1065467309 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7efe300f4750 0x7efe30106820 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:04.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 >> 172.21.15.76:0/1065467309 conn(0x7efe300f01d0 msgr2=0x7efe300f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:04.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 shutdown_connections 2023-12-26T22:03:04.448 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:04.447+0000 7efe34a0a700 1 -- 172.21.15.76:0/1065467309 wait complete. 2023-12-26T22:03:04.448 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:03.385300+0000 mgr.smithi059.gwkiyc (mgr.14156) 197 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:04.447026+0000 mon.smithi059 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.76:0/1065467309' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:05.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:05.827 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 -- 172.21.15.76:0/1933399519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 msgr2=0x7f50e40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 --2- 172.21.15.76:0/1933399519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e40f58a0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f50d4009fb0 tx=0x7f50d4009b30).stop 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 -- 172.21.15.76:0/1933399519 shutdown_connections 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 --2- 172.21.15.76:0/1933399519 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e40f58a0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 -- 172.21.15.76:0/1933399519 >> 172.21.15.76:0/1933399519 conn(0x7f50e40f0be0 msgr2=0x7f50e40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 -- 172.21.15.76:0/1933399519 shutdown_connections 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 -- 172.21.15.76:0/1933399519 wait complete. 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 Processor -- start 2023-12-26T22:03:06.999 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.995+0000 7f50ec10a700 1 -- start start 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50ec10a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e4109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50ec10a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50e4060300 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50e9ea6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e4109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50e9ea6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e4109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:38446/0 (socket says 172.21.15.76:38446) 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50e9ea6700 1 -- 172.21.15.76:0/2092737047 learned_addr learned my addr 172.21.15.76:0/2092737047 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50e9ea6700 1 -- 172.21.15.76:0/2092737047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50d4005040 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50e9ea6700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e4109510 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f50d40089f0 tx=0x7f50d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.996+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d4009320 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.997+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50e4060500 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.997+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f50d4013ec0 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.997+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50d4014d30 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.997+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50e4109bc0 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.997+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f50d40263a0 con 0x7f50e40f5480 2023-12-26T22:03:07.000 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.998+0000 7f50da7fc700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f50d0031090 0x7f50d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:07.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.998+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f50d4042450 con 0x7f50e40f5480 2023-12-26T22:03:07.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.998+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50e40591e0 con 0x7f50e40f5480 2023-12-26T22:03:07.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.998+0000 7f50e96a5700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f50d0031090 0x7f50d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:07.001 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:06.998+0000 7f50e96a5700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f50d0031090 0x7f50d0033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f50e00096a0 tx=0x7f50e0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:07.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.002+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f50d4012920 con 0x7f50e40f5480 2023-12-26T22:03:07.211 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.211+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50e40faf70 con 0x7f50e40f5480 2023-12-26T22:03:07.211 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.211+0000 7f50da7fc700 1 -- 172.21.15.76:0/2092737047 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f50d402b020 con 0x7f50e40f5480 2023-12-26T22:03:07.212 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:07.212 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f50d0031090 msgr2=0x7f50d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f50d0031090 0x7f50d0033550 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f50e00096a0 tx=0x7f50e0006b40).stop 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 msgr2=0x7f50e4109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e4109510 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f50d40089f0 tx=0x7f50d400b040).stop 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 shutdown_connections 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f50d0031090 0x7f50d0033550 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 --2- 172.21.15.76:0/2092737047 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f50e40f5480 0x7f50e4109510 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 >> 172.21.15.76:0/2092737047 conn(0x7f50e40f0be0 msgr2=0x7f50e40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:07.213 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 shutdown_connections 2023-12-26T22:03:07.214 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:07.213+0000 7f50ec10a700 1 -- 172.21.15.76:0/2092737047 wait complete. 2023-12-26T22:03:07.214 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:05.385589+0000 mgr.smithi059.gwkiyc (mgr.14156) 198 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:08.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:07 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:07.213194+0000 mon.smithi059 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.76:0/2092737047' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:08.615 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:08.616 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:09.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:07.385969+0000 mgr.smithi059.gwkiyc (mgr.14156) 199 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 -- 172.21.15.76:0/2565080815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 msgr2=0x7fe7a40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 --2- 172.21.15.76:0/2565080815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a40f58a0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7fe794009fb0 tx=0x7fe794009b30).stop 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 -- 172.21.15.76:0/2565080815 shutdown_connections 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 --2- 172.21.15.76:0/2565080815 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a40f58a0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 -- 172.21.15.76:0/2565080815 >> 172.21.15.76:0/2565080815 conn(0x7fe7a40f0be0 msgr2=0x7fe7a40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 -- 172.21.15.76:0/2565080815 shutdown_connections 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.790+0000 7fe7ac503700 1 -- 172.21.15.76:0/2565080815 wait complete. 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7ac503700 1 Processor -- start 2023-12-26T22:03:09.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7ac503700 1 -- start start 2023-12-26T22:03:09.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7ac503700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:09.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7ac503700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7a4106cb0 con 0x7fe7a40f5480 2023-12-26T22:03:09.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7aa29f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:09.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7aa29f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:38462/0 (socket says 172.21.15.76:38462) 2023-12-26T22:03:09.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.791+0000 7fe7aa29f700 1 -- 172.21.15.76:0/2497873041 learned_addr learned my addr 172.21.15.76:0/2497873041 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:09.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe7aa29f700 1 -- 172.21.15.76:0/2497873041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe794005040 con 0x7fe7a40f5480 2023-12-26T22:03:09.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe7aa29f700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a41067c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fe7940089f0 tx=0x7fe79400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:09.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe794009320 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7a4106eb0 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe794013ec0 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe794014d30 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.792+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7a4107340 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.793+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe794014420 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.793+0000 7fe79affd700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe790031090 0x7fe790033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.793+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7a4045bb0 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.793+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe7940424a0 con 0x7fe7a40f5480 2023-12-26T22:03:09.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.793+0000 7fe7a9a9e700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe790031090 0x7fe790033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:09.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.794+0000 7fe7a9a9e700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe790031090 0x7fe790033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe7a00096a0 tx=0x7fe7a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:09.798 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:09.798+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe79401d7f0 con 0x7fe7a40f5480 2023-12-26T22:03:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.006+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe7a41081f0 con 0x7fe7a40f5480 2023-12-26T22:03:10.006 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.006+0000 7fe79affd700 1 -- 172.21.15.76:0/2497873041 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe79401d7f0 con 0x7fe7a40f5480 2023-12-26T22:03:10.006 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:10.007 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe790031090 msgr2=0x7fe790033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe790031090 0x7fe790033550 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fe7a00096a0 tx=0x7fe7a0006b40).stop 2023-12-26T22:03:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 msgr2=0x7fe7a41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a41067c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7fe7940089f0 tx=0x7fe79400b040).stop 2023-12-26T22:03:10.008 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 shutdown_connections 2023-12-26T22:03:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe790031090 0x7fe790033550 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 --2- 172.21.15.76:0/2497873041 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe7a40f5480 0x7fe7a41067c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 >> 172.21.15.76:0/2497873041 conn(0x7fe7a40f0be0 msgr2=0x7fe7a405fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 shutdown_connections 2023-12-26T22:03:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:10.008+0000 7fe7ac503700 1 -- 172.21.15.76:0/2497873041 wait complete. 2023-12-26T22:03:10.009 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:09.386333+0000 mgr.smithi059.gwkiyc (mgr.14156) 200 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:10.008110+0000 mon.smithi059 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.76:0/2497873041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:11.416 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:11.416 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:12.557 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 -- 172.21.15.76:0/151883957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 msgr2=0x7f73640f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:12.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 --2- 172.21.15.76:0/151883957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73640f58a0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f7358009fb0 tx=0x7f7358009b30).stop 2023-12-26T22:03:12.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 -- 172.21.15.76:0/151883957 shutdown_connections 2023-12-26T22:03:12.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 --2- 172.21.15.76:0/151883957 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73640f58a0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:12.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 -- 172.21.15.76:0/151883957 >> 172.21.15.76:0/151883957 conn(0x7f73640f0be0 msgr2=0x7f73640f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:12.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 -- 172.21.15.76:0/151883957 shutdown_connections 2023-12-26T22:03:12.558 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.557+0000 7f73690eb700 1 -- 172.21.15.76:0/151883957 wait complete. 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.558+0000 7f73690eb700 1 Processor -- start 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.558+0000 7f73690eb700 1 -- start start 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.558+0000 7f73690eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73641094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.558+0000 7f73690eb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7364060320 con 0x7f73640f5480 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f7362d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73641094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f7362d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73641094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:38478/0 (socket says 172.21.15.76:38478) 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f7362d9d700 1 -- 172.21.15.76:0/3193769812 learned_addr learned my addr 172.21.15.76:0/3193769812 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f7362d9d700 1 -- 172.21.15.76:0/3193769812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7358005040 con 0x7f73640f5480 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f7362d9d700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73641094f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f73580089f0 tx=0x7f735800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:12.560 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7358009320 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7364060520 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7358013ec0 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7358014d30 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.559+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7364109ba0 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.560+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f73580263a0 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.560+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7364045bb0 con 0x7f73640f5480 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.560+0000 7f734b7fe700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f734c0310e0 0x7f734c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:12.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.560+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f7358042440 con 0x7f73640f5480 2023-12-26T22:03:12.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.560+0000 7f736259c700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f734c0310e0 0x7f734c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:12.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.561+0000 7f736259c700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f734c0310e0 0x7f734c0335a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f73540096a0 tx=0x7f7354006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:12.565 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.565+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7358012330 con 0x7f73640f5480 2023-12-26T22:03:12.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.775+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f73640faf70 con 0x7f73640f5480 2023-12-26T22:03:12.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.776+0000 7f734b7fe700 1 -- 172.21.15.76:0/3193769812 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f735801ccb0 con 0x7f73640f5480 2023-12-26T22:03:12.776 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:12.776 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:12.777 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f734c0310e0 msgr2=0x7f734c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:12.777 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f734c0310e0 0x7f734c0335a0 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f73540096a0 tx=0x7f7354006b40).stop 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 msgr2=0x7f73641094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73641094f0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f73580089f0 tx=0x7f735800b040).stop 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 shutdown_connections 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f734c0310e0 0x7f734c0335a0 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 --2- 172.21.15.76:0/3193769812 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f73640f5480 0x7f73641094f0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 >> 172.21.15.76:0/3193769812 conn(0x7f73640f0be0 msgr2=0x7f73640f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 shutdown_connections 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:12.777+0000 7f73690eb700 1 -- 172.21.15.76:0/3193769812 wait complete. 2023-12-26T22:03:12.778 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:13.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:11.386630+0000 mgr.smithi059.gwkiyc (mgr.14156) 201 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:13.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:12.777271+0000 mon.smithi059 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.76:0/3193769812' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:14.172 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:14.172 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:15.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:13.386948+0000 mgr.smithi059.gwkiyc (mgr.14156) 202 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:15.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 -- 172.21.15.76:0/1794169206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 msgr2=0x7f1db00f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:15.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 --2- 172.21.15.76:0/1794169206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db00f6800 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f1da0009fb0 tx=0x7f1da0009b30).stop 2023-12-26T22:03:15.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 -- 172.21.15.76:0/1794169206 shutdown_connections 2023-12-26T22:03:15.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 --2- 172.21.15.76:0/1794169206 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db00f6800 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:15.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 -- 172.21.15.76:0/1794169206 >> 172.21.15.76:0/1794169206 conn(0x7f1db00f1b40 msgr2=0x7f1db00f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:15.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 -- 172.21.15.76:0/1794169206 shutdown_connections 2023-12-26T22:03:15.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 -- 172.21.15.76:0/1794169206 wait complete. 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.339+0000 7f1db6665700 1 Processor -- start 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db6665700 1 -- start start 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db6665700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db0106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db6665700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1db0106b10 con 0x7f1db00f63e0 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db5663700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db0106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db5663700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db0106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:38502/0 (socket says 172.21.15.76:38502) 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db5663700 1 -- 172.21.15.76:0/1370002466 learned_addr learned my addr 172.21.15.76:0/1370002466 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db5663700 1 -- 172.21.15.76:0/1370002466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da0005040 con 0x7f1db00f63e0 2023-12-26T22:03:15.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.340+0000 7f1db5663700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db0106620 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f1da00089f0 tx=0x7f1da000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.341+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da0009320 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.341+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1db0106d10 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.341+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1da0013ec0 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.341+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1da0014d30 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.341+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1db01071a0 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.342+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1da00095f0 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.342+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1db0045bb0 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.342+0000 7f1da5ffb700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1d9c0310e0 0x7f1d9c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.342+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1da00423d0 con 0x7f1db00f63e0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.342+0000 7f1db4e62700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1d9c0310e0 0x7f1d9c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:15.343 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.343+0000 7f1db4e62700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1d9c0310e0 0x7f1d9c0335a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1dac0096a0 tx=0x7f1dac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:15.346 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.347+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1da001d7f0 con 0x7f1db00f63e0 2023-12-26T22:03:15.552 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.552+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1db0107ff0 con 0x7f1db00f63e0 2023-12-26T22:03:15.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.554+0000 7f1da5ffb700 1 -- 172.21.15.76:0/1370002466 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1da000fe40 con 0x7f1db00f63e0 2023-12-26T22:03:15.553 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:15.553 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:15.555 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1d9c0310e0 msgr2=0x7f1d9c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:15.555 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1d9c0310e0 0x7f1d9c0335a0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1dac0096a0 tx=0x7f1dac006b40).stop 2023-12-26T22:03:15.555 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 msgr2=0x7f1db0106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:15.555 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db0106620 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f1da00089f0 tx=0x7f1da000b040).stop 2023-12-26T22:03:15.555 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 shutdown_connections 2023-12-26T22:03:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1d9c0310e0 0x7f1d9c0335a0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 --2- 172.21.15.76:0/1370002466 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1db00f63e0 0x7f1db0106620 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 >> 172.21.15.76:0/1370002466 conn(0x7f1db00f1b40 msgr2=0x7f1db00f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 shutdown_connections 2023-12-26T22:03:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:15.555+0000 7f1db6665700 1 -- 172.21.15.76:0/1370002466 wait complete. 2023-12-26T22:03:15.556 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:16.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:15 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:15.554464+0000 mon.smithi059 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.76:0/1370002466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:16.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:16.946 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:17.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:15.387319+0000 mgr.smithi059.gwkiyc (mgr.14156) 203 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:18.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 -- 172.21.15.76:0/3182921961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 msgr2=0x7f303c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:18.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 --2- 172.21.15.76:0/3182921961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c0f69b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f302c004760 tx=0x7f302c009ad0).stop 2023-12-26T22:03:18.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 -- 172.21.15.76:0/3182921961 shutdown_connections 2023-12-26T22:03:18.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 --2- 172.21.15.76:0/3182921961 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c0f69b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:18.125 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 -- 172.21.15.76:0/3182921961 >> 172.21.15.76:0/3182921961 conn(0x7f303c0f1cd0 msgr2=0x7f303c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 -- 172.21.15.76:0/3182921961 shutdown_connections 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.124+0000 7f30446e6700 1 -- 172.21.15.76:0/3182921961 wait complete. 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f30446e6700 1 Processor -- start 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f30446e6700 1 -- start start 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f30446e6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c1067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f30446e6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f303c106cb0 con 0x7f303c0f6590 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f3042482700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c1067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f3042482700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c1067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47732/0 (socket says 172.21.15.76:47732) 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.125+0000 7f3042482700 1 -- 172.21.15.76:0/2903057186 learned_addr learned my addr 172.21.15.76:0/2903057186 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f3042482700 1 -- 172.21.15.76:0/2903057186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f302c009d70 con 0x7f303c0f6590 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f3042482700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c1067c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f302c004170 tx=0x7f302c009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302c00e040 con 0x7f303c0f6590 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f303c106eb0 con 0x7f303c0f6590 2023-12-26T22:03:18.126 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f302c008ef0 con 0x7f303c0f6590 2023-12-26T22:03:18.127 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f302c014c90 con 0x7f303c0f6590 2023-12-26T22:03:18.127 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.126+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f303c107340 con 0x7f303c0f6590 2023-12-26T22:03:18.127 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.127+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f302c014420 con 0x7f303c0f6590 2023-12-26T22:03:18.128 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.127+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f303c045bb0 con 0x7f303c0f6590 2023-12-26T22:03:18.128 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.127+0000 7f3032ffd700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3028031090 0x7f3028033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:18.128 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.127+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f302c042210 con 0x7f303c0f6590 2023-12-26T22:03:18.128 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.127+0000 7f3041c81700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3028031090 0x7f3028033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:18.128 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.127+0000 7f3041c81700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3028031090 0x7f3028033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f30380096a0 tx=0x7f3038006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:18.130 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.132+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f302c026300 con 0x7f303c0f6590 2023-12-26T22:03:18.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.338+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f303c108190 con 0x7f303c0f6590 2023-12-26T22:03:18.338 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.339+0000 7f3032ffd700 1 -- 172.21.15.76:0/2903057186 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f302c012b20 con 0x7f303c0f6590 2023-12-26T22:03:18.338 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:18.338 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:18.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3028031090 msgr2=0x7f3028033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3028031090 0x7f3028033550 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f30380096a0 tx=0x7f3038006b40).stop 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 msgr2=0x7f303c1067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c1067c0 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f302c004170 tx=0x7f302c009ad0).stop 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 shutdown_connections 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3028031090 0x7f3028033550 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 --2- 172.21.15.76:0/2903057186 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f303c0f6590 0x7f303c1067c0 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 >> 172.21.15.76:0/2903057186 conn(0x7f303c0f1cd0 msgr2=0x7f303c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 shutdown_connections 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:18.340+0000 7f30446e6700 1 -- 172.21.15.76:0/2903057186 wait complete. 2023-12-26T22:03:18.340 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:19.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:17.387702+0000 mgr.smithi059.gwkiyc (mgr.14156) 204 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:19.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:18.339516+0000 mon.smithi059 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.76:0/2903057186' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:19.721 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:19.721 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:20.916 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/471132698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 msgr2=0x7f1b840f4c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:20.916 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 --2- 172.21.15.76:0/471132698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b840f4c60 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f1b78009fb0 tx=0x7f1b78009b30).stop 2023-12-26T22:03:20.916 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/471132698 shutdown_connections 2023-12-26T22:03:20.916 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 --2- 172.21.15.76:0/471132698 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b840f4c60 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:20.916 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/471132698 >> 172.21.15.76:0/471132698 conn(0x7f1b840f0360 msgr2=0x7f1b840f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:20.917 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/471132698 shutdown_connections 2023-12-26T22:03:20.917 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/471132698 wait complete. 2023-12-26T22:03:20.917 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.917+0000 7f1b8a4d3700 1 Processor -- start 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b8a4d3700 1 -- start start 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b8a4d3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b841067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b8a4d3700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b84106cb0 con 0x7f1b840f4840 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b83fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b841067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b83fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b841067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47742/0 (socket says 172.21.15.76:47742) 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b83fff700 1 -- 172.21.15.76:0/2723991506 learned_addr learned my addr 172.21.15.76:0/2723991506 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b83fff700 1 -- 172.21.15.76:0/2723991506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b78005040 con 0x7f1b840f4840 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b83fff700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b841067c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1b780089f0 tx=0x7f1b7800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.918+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b78009320 con 0x7f1b840f4840 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b84106eb0 con 0x7f1b840f4840 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f1b78013ec0 con 0x7f1b840f4840 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1b78014d30 con 0x7f1b840f4840 2023-12-26T22:03:20.919 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b84107340 con 0x7f1b840f4840 2023-12-26T22:03:20.920 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f1b78014420 con 0x7f1b840f4840 2023-12-26T22:03:20.920 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b84045bb0 con 0x7f1b840f4840 2023-12-26T22:03:20.920 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.919+0000 7f1b80ff9700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1b6c031090 0x7f1b6c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:20.920 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.920+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f1b78042400 con 0x7f1b840f4840 2023-12-26T22:03:20.920 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.920+0000 7f1b837fe700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1b6c031090 0x7f1b6c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:20.920 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.920+0000 7f1b837fe700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1b6c031090 0x7f1b6c033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1b740096a0 tx=0x7f1b74006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:20.923 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:20.924+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f1b780268c0 con 0x7f1b840f4840 2023-12-26T22:03:21.129 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.130+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1b841081f0 con 0x7f1b840f4840 2023-12-26T22:03:21.130 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.131+0000 7f1b80ff9700 1 -- 172.21.15.76:0/2723991506 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f1b7801d380 con 0x7f1b840f4840 2023-12-26T22:03:21.130 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:21.130 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1b6c031090 msgr2=0x7f1b6c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1b6c031090 0x7f1b6c033550 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f1b740096a0 tx=0x7f1b74006b40).stop 2023-12-26T22:03:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 msgr2=0x7f1b841067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b841067c0 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f1b780089f0 tx=0x7f1b7800b040).stop 2023-12-26T22:03:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 shutdown_connections 2023-12-26T22:03:21.131 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1b6c031090 0x7f1b6c033550 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:21.132 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 --2- 172.21.15.76:0/2723991506 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f1b840f4840 0x7f1b841067c0 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:21.132 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 >> 172.21.15.76:0/2723991506 conn(0x7f1b840f0360 msgr2=0x7f1b840f0ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:21.132 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 shutdown_connections 2023-12-26T22:03:21.132 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:21.132+0000 7f1b8a4d3700 1 -- 172.21.15.76:0/2723991506 wait complete. 2023-12-26T22:03:21.132 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:19.388072+0000 mgr.smithi059.gwkiyc (mgr.14156) 205 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:22.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:21 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:21.131406+0000 mon.smithi059 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.76:0/2723991506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:22.520 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:22.520 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:21.388426+0000 mgr.smithi059.gwkiyc (mgr.14156) 206 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 -- 172.21.15.76:0/1729382335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 msgr2=0x7f13d80f4c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 --2- 172.21.15.76:0/1729382335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d80f4c70 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f13cc009fb0 tx=0x7f13cc009b30).stop 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 -- 172.21.15.76:0/1729382335 shutdown_connections 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 --2- 172.21.15.76:0/1729382335 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d80f4c70 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 -- 172.21.15.76:0/1729382335 >> 172.21.15.76:0/1729382335 conn(0x7f13d80f0390 msgr2=0x7f13d80f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 -- 172.21.15.76:0/1729382335 shutdown_connections 2023-12-26T22:03:23.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 -- 172.21.15.76:0/1729382335 wait complete. 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.722+0000 7f13df18f700 1 Processor -- start 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13df18f700 1 -- start start 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13df18f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d81067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13df18f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13d8106c90 con 0x7f13d80f4850 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13dcf2b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d81067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13dcf2b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d81067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47768/0 (socket says 172.21.15.76:47768) 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13dcf2b700 1 -- 172.21.15.76:0/3512480077 learned_addr learned my addr 172.21.15.76:0/3512480077 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.723+0000 7f13dcf2b700 1 -- 172.21.15.76:0/3512480077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13cc005040 con 0x7f13d80f4850 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13dcf2b700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d81067a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f13cc0089f0 tx=0x7f13cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:23.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13cc009320 con 0x7f13d80f4850 2023-12-26T22:03:23.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13d8106e90 con 0x7f13d80f4850 2023-12-26T22:03:23.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f13cc013ec0 con 0x7f13d80f4850 2023-12-26T22:03:23.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13cc014d30 con 0x7f13d80f4850 2023-12-26T22:03:23.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d8107320 con 0x7f13d80f4850 2023-12-26T22:03:23.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.724+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f13cc0095f0 con 0x7f13d80f4850 2023-12-26T22:03:23.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.725+0000 7f13d57fa700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f13c8031090 0x7f13c8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:23.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.725+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f13cc042420 con 0x7f13d80f4850 2023-12-26T22:03:23.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.725+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13d8045bb0 con 0x7f13d80f4850 2023-12-26T22:03:23.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.725+0000 7f13d7fff700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f13c8031090 0x7f13c8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:23.725 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.725+0000 7f13d7fff700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f13c8031090 0x7f13c8033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f13c40096a0 tx=0x7f13c4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:23.728 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.730+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f13cc01d7f0 con 0x7f13d80f4850 2023-12-26T22:03:23.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.939+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13d80fc080 con 0x7f13d80f4850 2023-12-26T22:03:23.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.940+0000 7f13d57fa700 1 -- 172.21.15.76:0/3512480077 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f13cc02b020 con 0x7f13d80f4850 2023-12-26T22:03:23.939 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:23.939 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f13c8031090 msgr2=0x7f13c8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f13c8031090 0x7f13c8033550 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f13c40096a0 tx=0x7f13c4006b40).stop 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 msgr2=0x7f13d81067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d81067a0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f13cc0089f0 tx=0x7f13cc00b040).stop 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 shutdown_connections 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f13c8031090 0x7f13c8033550 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 --2- 172.21.15.76:0/3512480077 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f13d80f4850 0x7f13d81067a0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 >> 172.21.15.76:0/3512480077 conn(0x7f13d80f0390 msgr2=0x7f13d80f0ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 shutdown_connections 2023-12-26T22:03:23.941 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:23.942+0000 7f13df18f700 1 -- 172.21.15.76:0/3512480077 wait complete. 2023-12-26T22:03:23.942 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:23.388782+0000 mgr.smithi059.gwkiyc (mgr.14156) 207 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:23.940759+0000 mon.smithi059 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.76:0/3512480077' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:25.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:25.328 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:26.539 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- 172.21.15.76:0/3988596535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 msgr2=0x7fe3000f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 --2- 172.21.15.76:0/3988596535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3000f58d0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fe2fc009fb0 tx=0x7fe2fc009b30).stop 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- 172.21.15.76:0/3988596535 shutdown_connections 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 --2- 172.21.15.76:0/3988596535 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3000f58d0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- 172.21.15.76:0/3988596535 >> 172.21.15.76:0/3988596535 conn(0x7fe3000f0c10 msgr2=0x7fe3000f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- 172.21.15.76:0/3988596535 shutdown_connections 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- 172.21.15.76:0/3988596535 wait complete. 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 Processor -- start 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- start start 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3001094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.540+0000 7fe308757700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe300060320 con 0x7fe3000f54b0 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe3064f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3001094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:26.540 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe3064f3700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3001094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44152/0 (socket says 172.21.15.76:44152) 2023-12-26T22:03:26.541 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe3064f3700 1 -- 172.21.15.76:0/2912403610 learned_addr learned my addr 172.21.15.76:0/2912403610 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe3064f3700 1 -- 172.21.15.76:0/2912403610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe2fc005040 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe3064f3700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3001094f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe2fc0089f0 tx=0x7fe2fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2fc009320 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe300060520 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe2fc013ec0 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe2fc014d30 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.541+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe300109ba0 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.542+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe2fc0095f0 con 0x7fe3000f54b0 2023-12-26T22:03:26.542 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.542+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe300045bb0 con 0x7fe3000f54b0 2023-12-26T22:03:26.543 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.542+0000 7fe2f6ffd700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2ec031090 0x7fe2ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:26.543 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.542+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe2fc042420 con 0x7fe3000f54b0 2023-12-26T22:03:26.543 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.543+0000 7fe305cf2700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2ec031090 0x7fe2ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:26.543 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.543+0000 7fe305cf2700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2ec031090 0x7fe2ec033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fe2f00096a0 tx=0x7fe2f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:26.546 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.547+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe2fc01d7f0 con 0x7fe3000f54b0 2023-12-26T22:03:26.756 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.757+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe3000fafa0 con 0x7fe3000f54b0 2023-12-26T22:03:26.756 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.758+0000 7fe2f6ffd700 1 -- 172.21.15.76:0/2912403610 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe2fc00fe40 con 0x7fe3000f54b0 2023-12-26T22:03:26.757 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:26.757 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:26.758 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2ec031090 msgr2=0x7fe2ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2ec031090 0x7fe2ec033550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fe2f00096a0 tx=0x7fe2f0006b40).stop 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 msgr2=0x7fe3001094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3001094f0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe2fc0089f0 tx=0x7fe2fc00b040).stop 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 shutdown_connections 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe2ec031090 0x7fe2ec033550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 --2- 172.21.15.76:0/2912403610 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe3000f54b0 0x7fe3001094f0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.759+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 >> 172.21.15.76:0/2912403610 conn(0x7fe3000f0c10 msgr2=0x7fe3000f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.760+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 shutdown_connections 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:26.760+0000 7fe308757700 1 -- 172.21.15.76:0/2912403610 wait complete. 2023-12-26T22:03:26.759 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:27.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:25.389063+0000 mgr.smithi059.gwkiyc (mgr.14156) 208 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:27.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:26.758434+0000 mon.smithi059 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.76:0/2912403610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:28.155 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:28.156 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:27.389326+0000 mgr.smithi059.gwkiyc (mgr.14156) 209 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:29.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 -- 172.21.15.76:0/4045716164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 msgr2=0x7f76340f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:29.336 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 --2- 172.21.15.76:0/4045716164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76340f69b0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f7624004760 tx=0x7f7624009ad0).stop 2023-12-26T22:03:29.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 -- 172.21.15.76:0/4045716164 shutdown_connections 2023-12-26T22:03:29.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 --2- 172.21.15.76:0/4045716164 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76340f69b0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:29.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 -- 172.21.15.76:0/4045716164 >> 172.21.15.76:0/4045716164 conn(0x7f76340f1cd0 msgr2=0x7f76340f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:29.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 -- 172.21.15.76:0/4045716164 shutdown_connections 2023-12-26T22:03:29.337 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.337+0000 7f763a342700 1 -- 172.21.15.76:0/4045716164 wait complete. 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.338+0000 7f763a342700 1 Processor -- start 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.338+0000 7f763a342700 1 -- start start 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.338+0000 7f763a342700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76341067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.338+0000 7f763a342700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7634106cb0 con 0x7f76340f6590 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7633fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76341067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7633fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76341067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44172/0 (socket says 172.21.15.76:44172) 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7633fff700 1 -- 172.21.15.76:0/3310582326 learned_addr learned my addr 172.21.15.76:0/3310582326 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7633fff700 1 -- 172.21.15.76:0/3310582326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7624009d70 con 0x7f76340f6590 2023-12-26T22:03:29.339 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7633fff700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76341067c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f7624013f10 tx=0x7f7624009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:29.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f762400e040 con 0x7f76340f6590 2023-12-26T22:03:29.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f7624007df0 con 0x7f76340f6590 2023-12-26T22:03:29.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7634106eb0 con 0x7f76340f6590 2023-12-26T22:03:29.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7624014d60 con 0x7f76340f6590 2023-12-26T22:03:29.340 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.339+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7634107340 con 0x7f76340f6590 2023-12-26T22:03:29.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.340+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f7624014520 con 0x7f76340f6590 2023-12-26T22:03:29.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.340+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7634045bb0 con 0x7f76340f6590 2023-12-26T22:03:29.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.340+0000 7f7630ff9700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f761c031090 0x7f761c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:29.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.341+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f76240424c0 con 0x7f76340f6590 2023-12-26T22:03:29.341 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.341+0000 7f76337fe700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f761c031090 0x7f761c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:29.342 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.341+0000 7f76337fe700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f761c031090 0x7f761c033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f76280096a0 tx=0x7f7628006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:29.344 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.345+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f7624007960 con 0x7f76340f6590 2023-12-26T22:03:29.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.553+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76341081f0 con 0x7f76340f6590 2023-12-26T22:03:29.552 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.553+0000 7f7630ff9700 1 -- 172.21.15.76:0/3310582326 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f76240262f0 con 0x7f76340f6590 2023-12-26T22:03:29.552 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:29.552 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f761c031090 msgr2=0x7f761c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f761c031090 0x7f761c033550 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f76280096a0 tx=0x7f7628006b40).stop 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 msgr2=0x7f76341067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76341067c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f7624013f10 tx=0x7f7624009ad0).stop 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 shutdown_connections 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f761c031090 0x7f761c033550 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 --2- 172.21.15.76:0/3310582326 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f76340f6590 0x7f76341067c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 >> 172.21.15.76:0/3310582326 conn(0x7f76340f1cd0 msgr2=0x7f76340f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 shutdown_connections 2023-12-26T22:03:29.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:29.555+0000 7f763a342700 1 -- 172.21.15.76:0/3310582326 wait complete. 2023-12-26T22:03:29.555 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:30.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:29 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:29.554020+0000 mon.smithi059 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.76:0/3310582326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:30.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:30.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:29.389686+0000 mgr.smithi059.gwkiyc (mgr.14156) 210 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 -- 172.21.15.76:0/3918604534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 msgr2=0x7f16780f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 --2- 172.21.15.76:0/3918604534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16780f58a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f166c009fb0 tx=0x7f166c009b30).stop 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 -- 172.21.15.76:0/3918604534 shutdown_connections 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 --2- 172.21.15.76:0/3918604534 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16780f58a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 -- 172.21.15.76:0/3918604534 >> 172.21.15.76:0/3918604534 conn(0x7f16780f0be0 msgr2=0x7f16780f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 -- 172.21.15.76:0/3918604534 shutdown_connections 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.151+0000 7f167cf84700 1 -- 172.21.15.76:0/3918604534 wait complete. 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167cf84700 1 Processor -- start 2023-12-26T22:03:32.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167cf84700 1 -- start start 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167cf84700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16781094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167cf84700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1678060320 con 0x7f16780f5480 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16781094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16781094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44176/0 (socket says 172.21.15.76:44176) 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167659c700 1 -- 172.21.15.76:0/2921887888 learned_addr learned my addr 172.21.15.76:0/2921887888 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.152+0000 7f167659c700 1 -- 172.21.15.76:0/2921887888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f166c005040 con 0x7f16780f5480 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.153+0000 7f167659c700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16781094f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f166c0089f0 tx=0x7f166c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.153+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166c009320 con 0x7f16780f5480 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.153+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1678060520 con 0x7f16780f5480 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.153+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f166c013ec0 con 0x7f16780f5480 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.153+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f166c014d30 con 0x7f16780f5480 2023-12-26T22:03:32.154 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.153+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1678109ba0 con 0x7f16780f5480 2023-12-26T22:03:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.154+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f166c0095f0 con 0x7f16780f5480 2023-12-26T22:03:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.154+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1678045bb0 con 0x7f16780f5480 2023-12-26T22:03:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.154+0000 7f1666ffd700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1660031090 0x7f1660033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.154+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f166c0423e0 con 0x7f16780f5480 2023-12-26T22:03:32.155 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.154+0000 7f1675d9b700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1660031090 0x7f1660033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:32.156 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.154+0000 7f1675d9b700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1660031090 0x7f1660033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f16680096a0 tx=0x7f1668006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:32.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.159+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f166c01d980 con 0x7f16780f5480 2023-12-26T22:03:32.364 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.364+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f16780faf70 con 0x7f16780f5480 2023-12-26T22:03:32.364 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.365+0000 7f1666ffd700 1 -- 172.21.15.76:0/2921887888 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f166c01d380 con 0x7f16780f5480 2023-12-26T22:03:32.365 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:32.365 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:32.366 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1660031090 msgr2=0x7f1660033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:32.366 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1660031090 0x7f1660033550 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f16680096a0 tx=0x7f1668006b40).stop 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 msgr2=0x7f16781094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16781094f0 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f166c0089f0 tx=0x7f166c00b040).stop 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 shutdown_connections 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f1660031090 0x7f1660033550 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 --2- 172.21.15.76:0/2921887888 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f16780f5480 0x7f16781094f0 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 >> 172.21.15.76:0/2921887888 conn(0x7f16780f0be0 msgr2=0x7f16780f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 shutdown_connections 2023-12-26T22:03:32.367 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:32.366+0000 7f167cf84700 1 -- 172.21.15.76:0/2921887888 wait complete. 2023-12-26T22:03:32.368 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:31.390058+0000 mgr.smithi059.gwkiyc (mgr.14156) 211 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:33.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:32.366296+0000 mon.smithi059 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.76:0/2921887888' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:33.745 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:33.745 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 -- 172.21.15.76:0/2304991053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 msgr2=0x7f38000f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 --2- 172.21.15.76:0/2304991053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f9050 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f37fc009fb0 tx=0x7f37fc009b30).stop 2023-12-26T22:03:34.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 -- 172.21.15.76:0/2304991053 shutdown_connections 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 --2- 172.21.15.76:0/2304991053 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f9050 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 -- 172.21.15.76:0/2304991053 >> 172.21.15.76:0/2304991053 conn(0x7f38000f0a30 msgr2=0x7f38000f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 -- 172.21.15.76:0/2304991053 shutdown_connections 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 -- 172.21.15.76:0/2304991053 wait complete. 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 Processor -- start 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 -- start start 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.970+0000 7f38063bb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f6ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:34.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38063bb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38000f4f50 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38053b9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f6ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38053b9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f6ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:44200/0 (socket says 172.21.15.76:44200) 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38053b9700 1 -- 172.21.15.76:0/1985679520 learned_addr learned my addr 172.21.15.76:0/1985679520 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38053b9700 1 -- 172.21.15.76:0/1985679520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37fc005040 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38053b9700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f6ad0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f37fc013fd0 tx=0x7f37fc009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37fc018070 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38000f50f0 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f37fc007b10 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.971+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37fc01c3c0 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.972+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38000f5580 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.973+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f37fc007c80 con 0x7f38000602c0 2023-12-26T22:03:34.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.973+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38000591e0 con 0x7f38000602c0 2023-12-26T22:03:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.973+0000 7f37f5ffb700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37ec031090 0x7f37ec033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.973+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f37fc042970 con 0x7f38000602c0 2023-12-26T22:03:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.973+0000 7f3804bb8700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37ec031090 0x7f37ec033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:34.974 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.973+0000 7f3804bb8700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37ec031090 0x7f37ec033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f37f00096a0 tx=0x7f37f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:34.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:34.978+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f37fc025b40 con 0x7f38000602c0 2023-12-26T22:03:35.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.185+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3800045bb0 con 0x7f38000602c0 2023-12-26T22:03:35.186 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.185+0000 7f37f5ffb700 1 -- 172.21.15.76:0/1985679520 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f37fc025b40 con 0x7f38000602c0 2023-12-26T22:03:35.186 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:35.186 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37ec031090 msgr2=0x7f37ec033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37ec031090 0x7f37ec033550 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f37f00096a0 tx=0x7f37f0006b40).stop 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 msgr2=0x7f38000f6ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f6ad0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f37fc013fd0 tx=0x7f37fc009b30).stop 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 shutdown_connections 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f37ec031090 0x7f37ec033550 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:35.187 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 --2- 172.21.15.76:0/1985679520 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f38000602c0 0x7f38000f6ad0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:35.188 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 >> 172.21.15.76:0/1985679520 conn(0x7f38000f0a30 msgr2=0x7f38000f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:35.188 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 shutdown_connections 2023-12-26T22:03:35.188 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:35.187+0000 7f38063bb700 1 -- 172.21.15.76:0/1985679520 wait complete. 2023-12-26T22:03:35.188 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:33.390430+0000 mgr.smithi059.gwkiyc (mgr.14156) 212 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:36.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:35 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:35.187219+0000 mon.smithi059 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.76:0/1985679520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:36.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:36.564 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:35.390715+0000 mgr.smithi059.gwkiyc (mgr.14156) 213 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 -- 172.21.15.76:0/1383117501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 msgr2=0x7f372c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 --2- 172.21.15.76:0/1383117501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0f58a0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f3720009fb0 tx=0x7f3720009b30).stop 2023-12-26T22:03:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 -- 172.21.15.76:0/1383117501 shutdown_connections 2023-12-26T22:03:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 --2- 172.21.15.76:0/1383117501 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0f58a0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 -- 172.21.15.76:0/1383117501 >> 172.21.15.76:0/1383117501 conn(0x7f372c0f0be0 msgr2=0x7f372c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:37.789 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 -- 172.21.15.76:0/1383117501 shutdown_connections 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.788+0000 7f3730f89700 1 -- 172.21.15.76:0/1383117501 wait complete. 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f3730f89700 1 Processor -- start 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f3730f89700 1 -- start start 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f3730f89700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f3730f89700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f372c0fb8e0 con 0x7f372c0f5480 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f372a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f372a59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58598/0 (socket says 172.21.15.76:58598) 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f372a59c700 1 -- 172.21.15.76:0/855170263 learned_addr learned my addr 172.21.15.76:0/855170263 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:37.790 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.789+0000 7f372a59c700 1 -- 172.21.15.76:0/855170263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3720005040 con 0x7f372c0f5480 2023-12-26T22:03:37.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.790+0000 7f372a59c700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0fd460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3720000c00 tx=0x7f372000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:37.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.790+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37200095f0 con 0x7f372c0f5480 2023-12-26T22:03:37.791 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.790+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f372c0fba80 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.790+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3720007e50 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.790+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3720014e70 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.790+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f372c0fbf10 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.791+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f37200146b0 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.791+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f372c0591e0 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.791+0000 7f371affd700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3714031090 0x7f3714033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.791+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3720041c50 con 0x7f372c0f5480 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.791+0000 7f3729d9b700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3714031090 0x7f3714033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:37.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.792+0000 7f3729d9b700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3714031090 0x7f3714033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f371c0096a0 tx=0x7f371c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:37.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:37.796+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f372001b2c0 con 0x7f372c0f5480 2023-12-26T22:03:38.003 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.002+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f372c0faf70 con 0x7f372c0f5480 2023-12-26T22:03:38.003 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.003+0000 7f371affd700 1 -- 172.21.15.76:0/855170263 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f372001b2c0 con 0x7f372c0f5480 2023-12-26T22:03:38.003 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:38.003 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3714031090 msgr2=0x7f3714033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3714031090 0x7f3714033550 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f371c0096a0 tx=0x7f371c006b40).stop 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 msgr2=0x7f372c0fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0fd460 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f3720000c00 tx=0x7f372000b040).stop 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 shutdown_connections 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3714031090 0x7f3714033550 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 --2- 172.21.15.76:0/855170263 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f372c0f5480 0x7f372c0fd460 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 >> 172.21.15.76:0/855170263 conn(0x7f372c0f0be0 msgr2=0x7f372c05fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 shutdown_connections 2023-12-26T22:03:38.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:38.005+0000 7f3730f89700 1 -- 172.21.15.76:0/855170263 wait complete. 2023-12-26T22:03:38.006 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:39.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:37.391012+0000 mgr.smithi059.gwkiyc (mgr.14156) 214 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:39.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:38.004812+0000 mon.smithi059 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.76:0/855170263' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:39.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:39.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 -- 172.21.15.76:0/4224918702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8700f5480 msgr2=0x7fb8700f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 --2- 172.21.15.76:0/4224918702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8700f5480 0x7fb8700f58a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fb86c009fb0 tx=0x7fb86c009b30).stop 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 -- 172.21.15.76:0/4224918702 shutdown_connections 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 --2- 172.21.15.76:0/4224918702 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb8700f5480 0x7fb8700f58a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 -- 172.21.15.76:0/4224918702 >> 172.21.15.76:0/4224918702 conn(0x7fb8700f0be0 msgr2=0x7fb8700f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 -- 172.21.15.76:0/4224918702 shutdown_connections 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.577+0000 7fb877a2d700 1 -- 172.21.15.76:0/4224918702 wait complete. 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb877a2d700 1 Processor -- start 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb877a2d700 1 -- start start 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb877a2d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 0x7fb8701095c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:40.579 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb877a2d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb870109ab0 con 0x7fb870060320 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb8757c9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 0x7fb8701095c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb8757c9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 0x7fb8701095c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58618/0 (socket says 172.21.15.76:58618) 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb8757c9700 1 -- 172.21.15.76:0/2110646624 learned_addr learned my addr 172.21.15.76:0/2110646624 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.578+0000 7fb8757c9700 1 -- 172.21.15.76:0/2110646624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb86c005040 con 0x7fb870060320 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.579+0000 7fb8757c9700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 0x7fb8701095c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fb86c009880 tx=0x7fb86c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.579+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb86c013680 con 0x7fb870060320 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.579+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb86c0095f0 con 0x7fb870060320 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.579+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb870109bf0 con 0x7fb870060320 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.579+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb86c014e40 con 0x7fb870060320 2023-12-26T22:03:40.580 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.579+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb87010a080 con 0x7fb870060320 2023-12-26T22:03:40.581 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.580+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb86c014420 con 0x7fb870060320 2023-12-26T22:03:40.581 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.580+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700591e0 con 0x7fb870060320 2023-12-26T22:03:40.581 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.580+0000 7fb865ffb700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb85c031090 0x7fb85c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:40.582 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.580+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb86c042430 con 0x7fb870060320 2023-12-26T22:03:40.582 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.580+0000 7fb874fc8700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb85c031090 0x7fb85c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:40.582 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.581+0000 7fb874fc8700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb85c031090 0x7fb85c033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb8600096a0 tx=0x7fb860006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:40.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.585+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb86c00e2a0 con 0x7fb870060320 2023-12-26T22:03:40.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.793+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb870045bb0 con 0x7fb870060320 2023-12-26T22:03:40.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.793+0000 7fb865ffb700 1 -- 172.21.15.76:0/2110646624 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb870045bb0 con 0x7fb870060320 2023-12-26T22:03:40.793 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:40.794 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb85c031090 msgr2=0x7fb85c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb85c031090 0x7fb85c033550 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fb8600096a0 tx=0x7fb860006b40).stop 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 msgr2=0x7fb8701095c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 0x7fb8701095c0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fb86c009880 tx=0x7fb86c009b30).stop 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 shutdown_connections 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb85c031090 0x7fb85c033550 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 --2- 172.21.15.76:0/2110646624 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb870060320 0x7fb8701095c0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:40.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 >> 172.21.15.76:0/2110646624 conn(0x7fb8700f0be0 msgr2=0x7fb8700f1760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:40.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 shutdown_connections 2023-12-26T22:03:40.796 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:40.795+0000 7fb877a2d700 1 -- 172.21.15.76:0/2110646624 wait complete. 2023-12-26T22:03:40.796 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:39.391310+0000 mgr.smithi059.gwkiyc (mgr.14156) 215 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:41.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:40.795053+0000 mon.smithi059 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.76:0/2110646624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:42.179 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:42.179 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:43.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:41.391552+0000 mgr.smithi059.gwkiyc (mgr.14156) 216 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:43.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.369+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1832677434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 msgr2=0x7fb5840f9210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:43.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.369+0000 7fb58c2ce700 1 --2- 172.21.15.76:0/1832677434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5840f9210 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7fb574009fb0 tx=0x7fb574009b30).stop 2023-12-26T22:03:43.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1832677434 shutdown_connections 2023-12-26T22:03:43.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 --2- 172.21.15.76:0/1832677434 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5840f9210 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:43.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1832677434 >> 172.21.15.76:0/1832677434 conn(0x7fb5840f0c10 msgr2=0x7fb5840f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1832677434 shutdown_connections 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1832677434 wait complete. 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 Processor -- start 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 -- start start 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5841067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58c2ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb584106c90 con 0x7fb5840602e0 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58a06a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5841067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58a06a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5841067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58632/0 (socket says 172.21.15.76:58632) 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.370+0000 7fb58a06a700 1 -- 172.21.15.76:0/1826289733 learned_addr learned my addr 172.21.15.76:0/1826289733 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb58a06a700 1 -- 172.21.15.76:0/1826289733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb574005040 con 0x7fb5840602e0 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb58a06a700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5841067a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fb574009f80 tx=0x7fb57400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb574009320 con 0x7fb5840602e0 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb584106e90 con 0x7fb5840602e0 2023-12-26T22:03:43.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb574013680 con 0x7fb5840602e0 2023-12-26T22:03:43.373 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb57401eda0 con 0x7fb5840602e0 2023-12-26T22:03:43.373 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.371+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb584107320 con 0x7fb5840602e0 2023-12-26T22:03:43.373 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.372+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb57401e420 con 0x7fb5840602e0 2023-12-26T22:03:43.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.372+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb584045bb0 con 0x7fb5840602e0 2023-12-26T22:03:43.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.372+0000 7fb57affd700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb5700310e0 0x7fb5700335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:43.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.372+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb574042420 con 0x7fb5840602e0 2023-12-26T22:03:43.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.372+0000 7fb589869700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb5700310e0 0x7fb5700335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:43.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.373+0000 7fb589869700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb5700310e0 0x7fb5700335a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb5800096a0 tx=0x7fb580006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:43.377 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.377+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb57401d380 con 0x7fb5840602e0 2023-12-26T22:03:43.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.585+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb5841081d0 con 0x7fb5840602e0 2023-12-26T22:03:43.585 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.586+0000 7fb57affd700 1 -- 172.21.15.76:0/1826289733 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb574004490 con 0x7fb5840602e0 2023-12-26T22:03:43.586 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:43.586 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:43.587 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.587+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb5700310e0 msgr2=0x7fb5700335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb5700310e0 0x7fb5700335a0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fb5800096a0 tx=0x7fb580006b40).stop 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 msgr2=0x7fb5841067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5841067a0 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fb574009f80 tx=0x7fb57400b040).stop 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 shutdown_connections 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb5700310e0 0x7fb5700335a0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 --2- 172.21.15.76:0/1826289733 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb5840602e0 0x7fb5841067a0 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 >> 172.21.15.76:0/1826289733 conn(0x7fb5840f0c10 msgr2=0x7fb5840f16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 shutdown_connections 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:43.588+0000 7fb58c2ce700 1 -- 172.21.15.76:0/1826289733 wait complete. 2023-12-26T22:03:43.588 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:44.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:43 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:43.587431+0000 mon.smithi059 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.76:0/1826289733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:44.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:44.984 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:45.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:43.391855+0000 mgr.smithi059.gwkiyc (mgr.14156) 217 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.261+0000 7f47a740b700 1 -- 172.21.15.76:0/3731746746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 msgr2=0x7f47a00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.261+0000 7f47a740b700 1 --2- 172.21.15.76:0/3731746746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a00f58a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f4794009fb0 tx=0x7f4794009b30).stop 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 -- 172.21.15.76:0/3731746746 shutdown_connections 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 --2- 172.21.15.76:0/3731746746 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a00f58a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 -- 172.21.15.76:0/3731746746 >> 172.21.15.76:0/3731746746 conn(0x7f47a00f0be0 msgr2=0x7f47a00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 -- 172.21.15.76:0/3731746746 shutdown_connections 2023-12-26T22:03:46.262 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 -- 172.21.15.76:0/3731746746 wait complete. 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 Processor -- start 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.262+0000 7f47a740b700 1 -- start start 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a740b700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a0109510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a740b700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47a0060300 con 0x7f47a00f5480 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a51a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a0109510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a51a7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a0109510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:49360/0 (socket says 172.21.15.76:49360) 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a51a7700 1 -- 172.21.15.76:0/203540349 learned_addr learned my addr 172.21.15.76:0/203540349 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a51a7700 1 -- 172.21.15.76:0/203540349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4794005040 con 0x7f47a00f5480 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f47a51a7700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a0109510 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f47940089f0 tx=0x7f479400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4794009320 con 0x7f47a00f5480 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4794013ec0 con 0x7f47a00f5480 2023-12-26T22:03:46.264 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.263+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4794014d30 con 0x7f47a00f5480 2023-12-26T22:03:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.264+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47a0060500 con 0x7f47a00f5480 2023-12-26T22:03:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.264+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47a0109bc0 con 0x7f47a00f5480 2023-12-26T22:03:46.265 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.265+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f47940095f0 con 0x7f47a00f5480 2023-12-26T22:03:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.265+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47a00591e0 con 0x7f47a00f5480 2023-12-26T22:03:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.265+0000 7f4791ffb700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f478c031090 0x7f478c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.265+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47940424a0 con 0x7f47a00f5480 2023-12-26T22:03:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.265+0000 7f47a49a6700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f478c031090 0x7f478c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:46.266 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.265+0000 7f47a49a6700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f478c031090 0x7f478c033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f479c0096a0 tx=0x7f479c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:46.269 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.269+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f479401d7f0 con 0x7f47a00f5480 2023-12-26T22:03:46.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:45.472180+0000 mon.smithi059 (mon.0) 272 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T22:03:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:45.472414+0000 mon.smithi059 (mon.0) 273 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T22:03:46.477 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.477+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47a0045bb0 con 0x7f47a00f5480 2023-12-26T22:03:46.477 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.478+0000 7f4791ffb700 1 -- 172.21.15.76:0/203540349 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f479401d7f0 con 0x7f47a00f5480 2023-12-26T22:03:46.478 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:46.478 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:46.479 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f478c031090 msgr2=0x7f478c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:46.479 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f478c031090 0x7f478c033550 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f479c0096a0 tx=0x7f479c006b40).stop 2023-12-26T22:03:46.479 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 msgr2=0x7f47a0109510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:46.479 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a0109510 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f47940089f0 tx=0x7f479400b040).stop 2023-12-26T22:03:46.479 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 shutdown_connections 2023-12-26T22:03:46.480 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f478c031090 0x7f478c033550 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:46.480 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 --2- 172.21.15.76:0/203540349 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47a00f5480 0x7f47a0109510 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:46.480 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 >> 172.21.15.76:0/203540349 conn(0x7f47a00f0be0 msgr2=0x7f47a00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:46.480 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 shutdown_connections 2023-12-26T22:03:46.480 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:46.479+0000 7f47a740b700 1 -- 172.21.15.76:0/203540349 wait complete. 2023-12-26T22:03:46.480 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:47.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:45.392202+0000 mgr.smithi059.gwkiyc (mgr.14156) 218 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:47.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:46.479348+0000 mon.smithi059 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.76:0/203540349' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:47.879 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:47.880 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:49.141 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 -- 172.21.15.76:0/3348511943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 msgr2=0x7f39b00f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:49.141 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 --2- 172.21.15.76:0/3348511943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b00f58a0 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f39a4009fb0 tx=0x7f39a4009b30).stop 2023-12-26T22:03:49.141 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 -- 172.21.15.76:0/3348511943 shutdown_connections 2023-12-26T22:03:49.141 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 --2- 172.21.15.76:0/3348511943 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b00f58a0 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:49.141 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 -- 172.21.15.76:0/3348511943 >> 172.21.15.76:0/3348511943 conn(0x7f39b00f0be0 msgr2=0x7f39b00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 -- 172.21.15.76:0/3348511943 shutdown_connections 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 -- 172.21.15.76:0/3348511943 wait complete. 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 Processor -- start 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 -- start start 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.141+0000 7f39b7d96700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b01094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b7d96700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39b0060320 con 0x7f39b00f5480 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b5b32700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b01094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b5b32700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b01094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:49382/0 (socket says 172.21.15.76:49382) 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b5b32700 1 -- 172.21.15.76:0/2690309954 learned_addr learned my addr 172.21.15.76:0/2690309954 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b5b32700 1 -- 172.21.15.76:0/2690309954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39a4005040 con 0x7f39b00f5480 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b5b32700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b01094f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f39a4000c00 tx=0x7f39a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:49.142 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39a40095f0 con 0x7f39b00f5480 2023-12-26T22:03:49.143 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f39a4007e50 con 0x7f39b00f5480 2023-12-26T22:03:49.143 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39b0060520 con 0x7f39b00f5480 2023-12-26T22:03:49.143 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39a4014e70 con 0x7f39b00f5480 2023-12-26T22:03:49.143 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.142+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39b0109ba0 con 0x7f39b00f5480 2023-12-26T22:03:49.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.143+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f39a4025070 con 0x7f39b00f5480 2023-12-26T22:03:49.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.143+0000 7f39a27fc700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f399c031090 0x7f399c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:49.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.143+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f39a4041e00 con 0x7f39b00f5480 2023-12-26T22:03:49.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.143+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39b0045bb0 con 0x7f39b00f5480 2023-12-26T22:03:49.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.144+0000 7f39b5331700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f399c031090 0x7f399c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:49.144 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.144+0000 7f39b5331700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f399c031090 0x7f399c033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f39ac0096a0 tx=0x7f39ac006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:49.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.148+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f39a40079c0 con 0x7f39b00f5480 2023-12-26T22:03:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:47.392536+0000 mgr.smithi059.gwkiyc (mgr.14156) 219 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:49.353 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.354+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f39b00fb060 con 0x7f39b00f5480 2023-12-26T22:03:49.354 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.355+0000 7f39a27fc700 1 -- 172.21.15.76:0/2690309954 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f39a400f300 con 0x7f39b00f5480 2023-12-26T22:03:49.354 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:49.354 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f399c031090 msgr2=0x7f399c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f399c031090 0x7f399c033550 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f39ac0096a0 tx=0x7f39ac006b40).stop 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 msgr2=0x7f39b01094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b01094f0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f39a4000c00 tx=0x7f39a400b040).stop 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 shutdown_connections 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f399c031090 0x7f399c033550 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 --2- 172.21.15.76:0/2690309954 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f39b00f5480 0x7f39b01094f0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 >> 172.21.15.76:0/2690309954 conn(0x7f39b00f0be0 msgr2=0x7f39b00f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:49.356 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 shutdown_connections 2023-12-26T22:03:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:49.356+0000 7f39b7d96700 1 -- 172.21.15.76:0/2690309954 wait complete. 2023-12-26T22:03:49.357 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:50.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:49 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:49.355486+0000 mon.smithi059 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.76:0/2690309954' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:50.725 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:50.725 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:51.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:49.392925+0000 mgr.smithi059.gwkiyc (mgr.14156) 220 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:51.894 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 -- 172.21.15.76:0/3225389371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 msgr2=0x7f94e40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:51.894 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 --2- 172.21.15.76:0/3225389371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e40f56f0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f94e0009fb0 tx=0x7f94e0009b30).stop 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 -- 172.21.15.76:0/3225389371 shutdown_connections 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 --2- 172.21.15.76:0/3225389371 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e40f56f0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 -- 172.21.15.76:0/3225389371 >> 172.21.15.76:0/3225389371 conn(0x7f94e40f0a30 msgr2=0x7f94e40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 -- 172.21.15.76:0/3225389371 shutdown_connections 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.895+0000 7f94eaaf2700 1 -- 172.21.15.76:0/3225389371 wait complete. 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94eaaf2700 1 Processor -- start 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94eaaf2700 1 -- start start 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94eaaf2700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e4109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:51.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94eaaf2700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f94e4060300 con 0x7f94e40f52d0 2023-12-26T22:03:51.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94e9af0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e4109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:51.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94e9af0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e4109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:49404/0 (socket says 172.21.15.76:49404) 2023-12-26T22:03:51.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94e9af0700 1 -- 172.21.15.76:0/1791104409 learned_addr learned my addr 172.21.15.76:0/1791104409 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94e9af0700 1 -- 172.21.15.76:0/1791104409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f94e0005040 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94e9af0700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e4109330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f94e00089f0 tx=0x7f94e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.896+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94e0009320 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.897+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f94e0013ec0 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.897+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f94e4060500 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.897+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f94e0014d30 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.897+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f94e41099e0 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.897+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f94e00095f0 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.898+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94e40591e0 con 0x7f94e40f52d0 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.898+0000 7f94da7fc700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f94d0031090 0x7f94d0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:51.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.898+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f94e00423e0 con 0x7f94e40f52d0 2023-12-26T22:03:51.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.898+0000 7f94e92ef700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f94d0031090 0x7f94d0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:51.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.898+0000 7f94e92ef700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f94d0031090 0x7f94d0033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f94d40096a0 tx=0x7f94d4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:51.901 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:51.903+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f94e001d7f0 con 0x7f94e40f52d0 2023-12-26T22:03:52.107 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.109+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f94e4045bb0 con 0x7f94e40f52d0 2023-12-26T22:03:52.108 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.109+0000 7f94da7fc700 1 -- 172.21.15.76:0/1791104409 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f94e001d7f0 con 0x7f94e40f52d0 2023-12-26T22:03:52.108 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:52.108 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:52.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f94d0031090 msgr2=0x7f94d0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:52.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f94d0031090 0x7f94d0033550 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f94d40096a0 tx=0x7f94d4006b40).stop 2023-12-26T22:03:52.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 msgr2=0x7f94e4109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:52.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e4109330 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f94e00089f0 tx=0x7f94e000b040).stop 2023-12-26T22:03:52.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 shutdown_connections 2023-12-26T22:03:52.110 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f94d0031090 0x7f94d0033550 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:52.111 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 --2- 172.21.15.76:0/1791104409 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f94e40f52d0 0x7f94e4109330 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:52.111 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 >> 172.21.15.76:0/1791104409 conn(0x7f94e40f0a30 msgr2=0x7f94e40f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:52.111 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 shutdown_connections 2023-12-26T22:03:52.111 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:52.111+0000 7f94eaaf2700 1 -- 172.21.15.76:0/1791104409 wait complete. 2023-12-26T22:03:52.111 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:53.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:51.393312+0000 mgr.smithi059.gwkiyc (mgr.14156) 221 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:53.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:52.109948+0000 mon.smithi059 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.76:0/1791104409' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:53.486 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:53.486 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 -- 172.21.15.76:0/3946097778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 msgr2=0x7eff340f9050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 --2- 172.21.15.76:0/3946097778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff340f9050 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7eff30009fb0 tx=0x7eff30009b30).stop 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 -- 172.21.15.76:0/3946097778 shutdown_connections 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 --2- 172.21.15.76:0/3946097778 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff340f9050 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 -- 172.21.15.76:0/3946097778 >> 172.21.15.76:0/3946097778 conn(0x7eff340f0a30 msgr2=0x7eff340f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 -- 172.21.15.76:0/3946097778 shutdown_connections 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 -- 172.21.15.76:0/3946097778 wait complete. 2023-12-26T22:03:54.679 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.679+0000 7eff3a3a1700 1 Processor -- start 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3a3a1700 1 -- start start 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3a3a1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff341065b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3a3a1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff34106aa0 con 0x7eff340602c0 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3939f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff341065b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3939f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff341065b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:49408/0 (socket says 172.21.15.76:49408) 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3939f700 1 -- 172.21.15.76:0/4110180892 learned_addr learned my addr 172.21.15.76:0/4110180892 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:54.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3939f700 1 -- 172.21.15.76:0/4110180892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff30005040 con 0x7eff340602c0 2023-12-26T22:03:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.680+0000 7eff3939f700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff341065b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7eff30009d00 tx=0x7eff30009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.681+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff30018070 con 0x7eff340602c0 2023-12-26T22:03:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.681+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff34106ca0 con 0x7eff340602c0 2023-12-26T22:03:54.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.681+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7eff3001d420 con 0x7eff340602c0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.681+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff3001c5a0 con 0x7eff340602c0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.681+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff34107130 con 0x7eff340602c0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.682+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7eff3001c860 con 0x7eff340602c0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.682+0000 7eff29ffb700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7eff20031090 0x7eff20033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.682+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7eff30042870 con 0x7eff340602c0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.682+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff34045bb0 con 0x7eff340602c0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.682+0000 7eff38b9e700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7eff20031090 0x7eff20033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:54.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.682+0000 7eff38b9e700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7eff20031090 0x7eff20033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7eff240096a0 tx=0x7eff24006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:54.685 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.687+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7eff30027e00 con 0x7eff340602c0 2023-12-26T22:03:54.895 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.896+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff340fbec0 con 0x7eff340602c0 2023-12-26T22:03:54.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.897+0000 7eff29ffb700 1 -- 172.21.15.76:0/4110180892 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7eff30027e00 con 0x7eff340602c0 2023-12-26T22:03:54.896 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:54.896 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:54.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7eff20031090 msgr2=0x7eff20033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7eff20031090 0x7eff20033550 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7eff240096a0 tx=0x7eff24006b40).stop 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 msgr2=0x7eff341065b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff341065b0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7eff30009d00 tx=0x7eff30009b30).stop 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 shutdown_connections 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7eff20031090 0x7eff20033550 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 --2- 172.21.15.76:0/4110180892 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7eff340602c0 0x7eff341065b0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 >> 172.21.15.76:0/4110180892 conn(0x7eff340f0a30 msgr2=0x7eff340f1520 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 shutdown_connections 2023-12-26T22:03:54.898 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:54.899+0000 7eff3a3a1700 1 -- 172.21.15.76:0/4110180892 wait complete. 2023-12-26T22:03:54.899 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:55.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:53.393649+0000 mgr.smithi059.gwkiyc (mgr.14156) 222 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:55.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:54.897760+0000 mon.smithi059 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.76:0/4110180892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:56.292 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:56.293 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:57.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:55.394010+0000 mgr.smithi059.gwkiyc (mgr.14156) 223 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:03:57.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 -- 172.21.15.76:0/434421240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 msgr2=0x7f20740f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:57.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 --2- 172.21.15.76:0/434421240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20740f58a0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f2070009fb0 tx=0x7f2070009b30).stop 2023-12-26T22:03:57.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 -- 172.21.15.76:0/434421240 shutdown_connections 2023-12-26T22:03:57.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 --2- 172.21.15.76:0/434421240 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20740f58a0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:57.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 -- 172.21.15.76:0/434421240 >> 172.21.15.76:0/434421240 conn(0x7f20740f0be0 msgr2=0x7f20740f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:57.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 -- 172.21.15.76:0/434421240 shutdown_connections 2023-12-26T22:03:57.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.502+0000 7f207c655700 1 -- 172.21.15.76:0/434421240 wait complete. 2023-12-26T22:03:57.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207c655700 1 Processor -- start 2023-12-26T22:03:57.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207c655700 1 -- start start 2023-12-26T22:03:57.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207c655700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20741089e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207c655700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2074108ed0 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207a3f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20741089e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207a3f1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20741089e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39448/0 (socket says 172.21.15.76:39448) 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207a3f1700 1 -- 172.21.15.76:0/3237020682 learned_addr learned my addr 172.21.15.76:0/3237020682 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207a3f1700 1 -- 172.21.15.76:0/3237020682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2070005040 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.503+0000 7f207a3f1700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20741089e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f20700089f0 tx=0x7f207000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.504+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2070009320 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.504+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20741090d0 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.504+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2070013ec0 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.504+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2070014d30 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.504+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2074109560 con 0x7f20740f5480 2023-12-26T22:03:57.505 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.504+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f20700095f0 con 0x7f20740f5480 2023-12-26T22:03:57.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.505+0000 7f206affd700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2060031090 0x7f2060033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:03:57.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.505+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f20700424a0 con 0x7f20740f5480 2023-12-26T22:03:57.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.505+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2074045bb0 con 0x7f20740f5480 2023-12-26T22:03:57.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.505+0000 7f2079bf0700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2060031090 0x7f2060033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:03:57.506 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.505+0000 7f2079bf0700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2060031090 0x7f2060033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f20640096a0 tx=0x7f2064006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:03:57.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.509+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f207001dd70 con 0x7f20740f5480 2023-12-26T22:03:57.718 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.719+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f207410a2b0 con 0x7f20740f5480 2023-12-26T22:03:57.719 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.720+0000 7f206affd700 1 -- 172.21.15.76:0/3237020682 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f207002b020 con 0x7f20740f5480 2023-12-26T22:03:57.719 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:03:57.719 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2060031090 msgr2=0x7f2060033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2060031090 0x7f2060033550 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f20640096a0 tx=0x7f2064006b40).stop 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 msgr2=0x7f20741089e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20741089e0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f20700089f0 tx=0x7f207000b040).stop 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 shutdown_connections 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2060031090 0x7f2060033550 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 --2- 172.21.15.76:0/3237020682 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20740f5480 0x7f20741089e0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 >> 172.21.15.76:0/3237020682 conn(0x7f20740f0be0 msgr2=0x7f207405ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 shutdown_connections 2023-12-26T22:03:57.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:03:57.722+0000 7f207c655700 1 -- 172.21.15.76:0/3237020682 wait complete. 2023-12-26T22:03:57.722 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:03:58.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:57 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:03:57.720836+0000 mon.smithi059 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.76:0/3237020682' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:03:59.106 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:03:59.106 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:03:59.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:03:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:57.394382+0000 mgr.smithi059.gwkiyc (mgr.14156) 224 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 -- 172.21.15.76:0/4199251753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 msgr2=0x7f83240f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 --2- 172.21.15.76:0/4199251753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f83240f58a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f830c009fb0 tx=0x7f830c009b30).stop 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 -- 172.21.15.76:0/4199251753 shutdown_connections 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 --2- 172.21.15.76:0/4199251753 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f83240f58a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 -- 172.21.15.76:0/4199251753 >> 172.21.15.76:0/4199251753 conn(0x7f83240f0be0 msgr2=0x7f83240f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 -- 172.21.15.76:0/4199251753 shutdown_connections 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.284+0000 7f8328d91700 1 -- 172.21.15.76:0/4199251753 wait complete. 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f8328d91700 1 Processor -- start 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f8328d91700 1 -- start start 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f8328d91700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f832410b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:00.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f8328d91700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8324107300 con 0x7f83240f5480 2023-12-26T22:04:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f832259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f832410b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f832259c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f832410b910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39464/0 (socket says 172.21.15.76:39464) 2023-12-26T22:04:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.285+0000 7f832259c700 1 -- 172.21.15.76:0/3776952389 learned_addr learned my addr 172.21.15.76:0/3776952389 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f832259c700 1 -- 172.21.15.76:0/3776952389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f830c005040 con 0x7f83240f5480 2023-12-26T22:04:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f832259c700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f832410b910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f830c0089f0 tx=0x7f830c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:00.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f830c009320 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8324060300 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f830c013ec0 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f830c014d30 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.286+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f832410be00 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.287+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f830c0263a0 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.287+0000 7f831affd700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8310031090 0x7f8310033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.287+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f830c0423a0 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.287+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8324045bb0 con 0x7f83240f5480 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.287+0000 7f8321d9b700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8310031090 0x7f8310033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:00.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.288+0000 7f8321d9b700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8310031090 0x7f8310033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f83140096a0 tx=0x7f8314006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:00.290 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.292+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f830c014420 con 0x7f83240f5480 2023-12-26T22:04:00.500 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.501+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83240fb060 con 0x7f83240f5480 2023-12-26T22:04:00.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.502+0000 7f831affd700 1 -- 172.21.15.76:0/3776952389 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f830c02b020 con 0x7f83240f5480 2023-12-26T22:04:00.501 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:00.502 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8310031090 msgr2=0x7f8310033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8310031090 0x7f8310033550 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f83140096a0 tx=0x7f8314006b40).stop 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 msgr2=0x7f832410b910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f832410b910 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f830c0089f0 tx=0x7f830c00b040).stop 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 shutdown_connections 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8310031090 0x7f8310033550 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:00.503 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 --2- 172.21.15.76:0/3776952389 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f83240f5480 0x7f832410b910 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:00.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 >> 172.21.15.76:0/3776952389 conn(0x7f83240f0be0 msgr2=0x7f83240f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:00.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 shutdown_connections 2023-12-26T22:04:00.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:00.504+0000 7f8328d91700 1 -- 172.21.15.76:0/3776952389 wait complete. 2023-12-26T22:04:00.504 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:03:59.394656+0000 mgr.smithi059.gwkiyc (mgr.14156) 225 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:01.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:00.502860+0000 mon.smithi059 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.76:0/3776952389' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:01.899 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:01.899 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.113+0000 7f9e3759e700 1 -- 172.21.15.76:0/2495832946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 msgr2=0x7f9e380f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.113+0000 7f9e3759e700 1 --2- 172.21.15.76:0/2495832946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380f56f0 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f9e28009fb0 tx=0x7f9e28009b30).stop 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.113+0000 7f9e3759e700 1 -- 172.21.15.76:0/2495832946 shutdown_connections 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.113+0000 7f9e3759e700 1 --2- 172.21.15.76:0/2495832946 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380f56f0 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.113+0000 7f9e3759e700 1 -- 172.21.15.76:0/2495832946 >> 172.21.15.76:0/2495832946 conn(0x7f9e380f0a10 msgr2=0x7f9e380f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3759e700 1 -- 172.21.15.76:0/2495832946 shutdown_connections 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3759e700 1 -- 172.21.15.76:0/2495832946 wait complete. 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3759e700 1 Processor -- start 2023-12-26T22:04:03.115 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3759e700 1 -- start start 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3759e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3759e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e380fb730 con 0x7f9e380f52d0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3659c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39486/0 (socket says 172.21.15.76:39486) 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.114+0000 7f9e3659c700 1 -- 172.21.15.76:0/1613309427 learned_addr learned my addr 172.21.15.76:0/1613309427 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e3659c700 1 -- 172.21.15.76:0/1613309427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e28005040 con 0x7f9e380f52d0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e3659c700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380fd2b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f9e280089f0 tx=0x7f9e2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e28009320 con 0x7f9e380f52d0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9e28013ec0 con 0x7f9e380f52d0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e380fb8d0 con 0x7f9e380f52d0 2023-12-26T22:04:03.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e28014d30 con 0x7f9e380f52d0 2023-12-26T22:04:03.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.115+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e380fbd60 con 0x7f9e380f52d0 2023-12-26T22:04:03.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.116+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f9e2801d380 con 0x7f9e380f52d0 2023-12-26T22:04:03.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.116+0000 7f9e26ffd700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e20031090 0x7f9e20033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:03.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.116+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9e2800f430 con 0x7f9e380f52d0 2023-12-26T22:04:03.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.117+0000 7f9e35d9b700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e20031090 0x7f9e20033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:03.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.117+0000 7f9e35d9b700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e20031090 0x7f9e20033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9e2c0096a0 tx=0x7f9e2c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:03.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.117+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e380591e0 con 0x7f9e380f52d0 2023-12-26T22:04:03.121 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.122+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f9e28048060 con 0x7f9e380f52d0 2023-12-26T22:04:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:01.394952+0000 mgr.smithi059.gwkiyc (mgr.14156) 226 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.327+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9e380fc3c0 con 0x7f9e380f52d0 2023-12-26T22:04:03.327 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.327+0000 7f9e26ffd700 1 -- 172.21.15.76:0/1613309427 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f9e2801bca0 con 0x7f9e380f52d0 2023-12-26T22:04:03.327 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:03.327 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:03.328 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e20031090 msgr2=0x7f9e20033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e20031090 0x7f9e20033550 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f9e2c0096a0 tx=0x7f9e2c006b40).stop 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 msgr2=0x7f9e380fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380fd2b0 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f9e280089f0 tx=0x7f9e2800b040).stop 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 shutdown_connections 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9e20031090 0x7f9e20033550 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 --2- 172.21.15.76:0/1613309427 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f9e380f52d0 0x7f9e380fd2b0 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 >> 172.21.15.76:0/1613309427 conn(0x7f9e380f0a10 msgr2=0x7f9e3805ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 shutdown_connections 2023-12-26T22:04:03.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:03.329+0000 7f9e3759e700 1 -- 172.21.15.76:0/1613309427 wait complete. 2023-12-26T22:04:03.330 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:04.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:03 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:03.329012+0000 mon.smithi059 (mon.0) 280 : audit [DBG] from='client.? 172.21.15.76:0/1613309427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:04.708 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:04.708 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:05.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:03.395281+0000 mgr.smithi059.gwkiyc (mgr.14156) 227 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:05.935 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.935+0000 7f5a8203d700 1 -- 172.21.15.76:0/4251153680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 msgr2=0x7f5a7c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:05.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.935+0000 7f5a8203d700 1 --2- 172.21.15.76:0/4251153680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c0f69b0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f5a64009fb0 tx=0x7f5a64009b30).stop 2023-12-26T22:04:05.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.935+0000 7f5a8203d700 1 -- 172.21.15.76:0/4251153680 shutdown_connections 2023-12-26T22:04:05.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.935+0000 7f5a8203d700 1 --2- 172.21.15.76:0/4251153680 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c0f69b0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:05.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.935+0000 7f5a8203d700 1 -- 172.21.15.76:0/4251153680 >> 172.21.15.76:0/4251153680 conn(0x7f5a7c0f1cf0 msgr2=0x7f5a7c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:05.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.936+0000 7f5a8203d700 1 -- 172.21.15.76:0/4251153680 shutdown_connections 2023-12-26T22:04:05.936 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.936+0000 7f5a8203d700 1 -- 172.21.15.76:0/4251153680 wait complete. 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.936+0000 7f5a8203d700 1 Processor -- start 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.936+0000 7f5a8203d700 1 -- start start 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a8203d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c1067d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a8203d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c106cc0 con 0x7f5a7c0f6590 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a7b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c1067d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a7b7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c1067d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58184/0 (socket says 172.21.15.76:58184) 2023-12-26T22:04:05.937 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/2778073410 learned_addr learned my addr 172.21.15.76:0/2778073410 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:05.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a7b7fe700 1 -- 172.21.15.76:0/2778073410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a64005040 con 0x7f5a7c0f6590 2023-12-26T22:04:05.938 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.937+0000 7f5a7b7fe700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c1067d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f5a640089f0 tx=0x7f5a6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.938+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a64009320 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.938+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c106ec0 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.938+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c107350 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.938+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5a64013ec0 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.938+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5a64014d00 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.939+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5a640263a0 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.939+0000 7f5a73fff700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5a68031090 0x7f5a68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.939+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5a64014420 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.939+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c0591e0 con 0x7f5a7c0f6590 2023-12-26T22:04:05.939 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.939+0000 7f5a7affd700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5a68031090 0x7f5a68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:05.940 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.940+0000 7f5a7affd700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5a68031090 0x7f5a68033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5a6c0096a0 tx=0x7f5a6c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:05.944 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:05.944+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f5a64012920 con 0x7f5a7c0f6590 2023-12-26T22:04:06.147 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.147+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5a7c0fc080 con 0x7f5a7c0f6590 2023-12-26T22:04:06.148 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.148+0000 7f5a73fff700 1 -- 172.21.15.76:0/2778073410 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f5a6402b020 con 0x7f5a7c0f6590 2023-12-26T22:04:06.148 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:06.148 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:06.149 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.149+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5a68031090 msgr2=0x7f5a68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.149+0000 7f5a8203d700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5a68031090 0x7f5a68033550 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5a6c0096a0 tx=0x7f5a6c006b40).stop 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.149+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 msgr2=0x7f5a7c1067d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.149+0000 7f5a8203d700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c1067d0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f5a640089f0 tx=0x7f5a6400b040).stop 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.150+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 shutdown_connections 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.150+0000 7f5a8203d700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f5a68031090 0x7f5a68033550 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.150+0000 7f5a8203d700 1 --2- 172.21.15.76:0/2778073410 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5a7c0f6590 0x7f5a7c1067d0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.150+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 >> 172.21.15.76:0/2778073410 conn(0x7f5a7c0f1cf0 msgr2=0x7f5a7c0f2770 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.150+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 shutdown_connections 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:06.150+0000 7f5a8203d700 1 -- 172.21.15.76:0/2778073410 wait complete. 2023-12-26T22:04:06.150 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:07.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:05.395544+0000 mgr.smithi059.gwkiyc (mgr.14156) 228 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:07.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:06.149779+0000 mon.smithi059 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.76:0/2778073410' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:07.539 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:07.539 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- 172.21.15.76:0/3409470945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 msgr2=0x7f01e80f4d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 --2- 172.21.15.76:0/3409470945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e80f4d20 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f01e4009fb0 tx=0x7f01e4009b30).stop 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- 172.21.15.76:0/3409470945 shutdown_connections 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 --2- 172.21.15.76:0/3409470945 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e80f4d20 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- 172.21.15.76:0/3409470945 >> 172.21.15.76:0/3409470945 conn(0x7f01e80f0360 msgr2=0x7f01e80f2780 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- 172.21.15.76:0/3409470945 shutdown_connections 2023-12-26T22:04:08.680 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- 172.21.15.76:0/3409470945 wait complete. 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 Processor -- start 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- start start 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e81067a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.679+0000 7f01f03ce700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01e8106c90 con 0x7f01e80f4900 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01ee16a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e81067a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01ee16a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e81067a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58208/0 (socket says 172.21.15.76:58208) 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01ee16a700 1 -- 172.21.15.76:0/3483416394 learned_addr learned my addr 172.21.15.76:0/3483416394 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01ee16a700 1 -- 172.21.15.76:0/3483416394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01e4005040 con 0x7f01e80f4900 2023-12-26T22:04:08.681 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01ee16a700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e81067a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f01e4009f80 tx=0x7f01e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:08.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e4009320 con 0x7f01e80f4900 2023-12-26T22:04:08.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01e8106e90 con 0x7f01e80f4900 2023-12-26T22:04:08.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f01e4013680 con 0x7f01e80f4900 2023-12-26T22:04:08.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f01e401eda0 con 0x7f01e80f4900 2023-12-26T22:04:08.682 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.680+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01e8107320 con 0x7f01e80f4900 2023-12-26T22:04:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.681+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f01e401e420 con 0x7f01e80f4900 2023-12-26T22:04:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.681+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01e8045bb0 con 0x7f01e80f4900 2023-12-26T22:04:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.681+0000 7f01deffd700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f01d4031090 0x7f01d4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.681+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f01e40423b0 con 0x7f01e80f4900 2023-12-26T22:04:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.682+0000 7f01ed969700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f01d4031090 0x7f01d4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:08.683 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.682+0000 7f01ed969700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f01d4031090 0x7f01d4033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f01d80096a0 tx=0x7f01d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:08.686 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.686+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f01e401cb10 con 0x7f01e80f4900 2023-12-26T22:04:08.893 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.893+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f01e8108170 con 0x7f01e80f4900 2023-12-26T22:04:08.894 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.894+0000 7f01deffd700 1 -- 172.21.15.76:0/3483416394 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f01e401cb10 con 0x7f01e80f4900 2023-12-26T22:04:08.894 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:08.895 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f01d4031090 msgr2=0x7f01d4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f01d4031090 0x7f01d4033550 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f01d80096a0 tx=0x7f01d8006b40).stop 2023-12-26T22:04:08.896 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 msgr2=0x7f01e81067a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e81067a0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f01e4009f80 tx=0x7f01e400b040).stop 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 shutdown_connections 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f01d4031090 0x7f01d4033550 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 --2- 172.21.15.76:0/3483416394 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f01e80f4900 0x7f01e81067a0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 >> 172.21.15.76:0/3483416394 conn(0x7f01e80f0360 msgr2=0x7f01e80f0eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 shutdown_connections 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:08.896+0000 7f01f03ce700 1 -- 172.21.15.76:0/3483416394 wait complete. 2023-12-26T22:04:08.897 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:07.395923+0000 mgr.smithi059.gwkiyc (mgr.14156) 229 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:09.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:08.895891+0000 mon.smithi059 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.76:0/3483416394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:10.279 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:10.279 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:11.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:09.396284+0000 mgr.smithi059.gwkiyc (mgr.14156) 230 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:11.482 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 -- 172.21.15.76:0/2057956393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 msgr2=0x7f3ad80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 --2- 172.21.15.76:0/2057956393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad80f58a0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f3ac8009fb0 tx=0x7f3ac8009b30).stop 2023-12-26T22:04:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 -- 172.21.15.76:0/2057956393 shutdown_connections 2023-12-26T22:04:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 --2- 172.21.15.76:0/2057956393 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad80f58a0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 -- 172.21.15.76:0/2057956393 >> 172.21.15.76:0/2057956393 conn(0x7f3ad80f0be0 msgr2=0x7f3ad80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:11.561 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 -- 172.21.15.76:0/2057956393 shutdown_connections 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.478+0000 7f3adfa50700 1 -- 172.21.15.76:0/2057956393 wait complete. 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3adfa50700 1 Processor -- start 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3adfa50700 1 -- start start 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3adfa50700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad81056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3adfa50700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ad8105be0 con 0x7f3ad80f5480 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3add7ec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad81056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3add7ec700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad81056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58228/0 (socket says 172.21.15.76:58228) 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3add7ec700 1 -- 172.21.15.76:0/205061766 learned_addr learned my addr 172.21.15.76:0/205061766 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:11.562 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3add7ec700 1 -- 172.21.15.76:0/205061766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ac8005040 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.479+0000 7f3add7ec700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad81056f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3ac80089f0 tx=0x7f3ac800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.480+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ac8009320 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.480+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ad8105de0 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.480+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3ac8013ec0 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.480+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ac8014d30 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.480+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ad8106270 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.481+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3ac80095f0 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.481+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ad8045bb0 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.481+0000 7f3acdffb700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3ac4031090 0x7f3ac4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.481+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3ac80424a0 con 0x7f3ad80f5480 2023-12-26T22:04:11.563 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.481+0000 7f3adcfeb700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3ac4031090 0x7f3ac4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:11.564 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.482+0000 7f3adcfeb700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3ac4031090 0x7f3ac4033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3ad40096a0 tx=0x7f3ad4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:11.564 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.486+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3ac801d7f0 con 0x7f3ad80f5480 2023-12-26T22:04:11.694 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.694+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ad80faf70 con 0x7f3ad80f5480 2023-12-26T22:04:11.697 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.695+0000 7f3acdffb700 1 -- 172.21.15.76:0/205061766 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3ac801d7f0 con 0x7f3ad80f5480 2023-12-26T22:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:11.697 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3ac4031090 msgr2=0x7f3ac4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3ac4031090 0x7f3ac4033550 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f3ad40096a0 tx=0x7f3ad4006b40).stop 2023-12-26T22:04:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 msgr2=0x7f3ad81056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:11.699 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad81056f0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f3ac80089f0 tx=0x7f3ac800b040).stop 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 shutdown_connections 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3ac4031090 0x7f3ac4033550 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 --2- 172.21.15.76:0/205061766 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3ad80f5480 0x7f3ad81056f0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 >> 172.21.15.76:0/205061766 conn(0x7f3ad80f0be0 msgr2=0x7f3ad80f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 shutdown_connections 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:11.696+0000 7f3adfa50700 1 -- 172.21.15.76:0/205061766 wait complete. 2023-12-26T22:04:11.700 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:12.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:11 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:11.696285+0000 mon.smithi059 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.76:0/205061766' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:13.069 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:13.070 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:13.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:11.396643+0000 mgr.smithi059.gwkiyc (mgr.14156) 231 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 -- 172.21.15.76:0/762926642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 msgr2=0x7fb33c0f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 --2- 172.21.15.76:0/762926642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0f56f0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7fb32c009fb0 tx=0x7fb32c009b30).stop 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 -- 172.21.15.76:0/762926642 shutdown_connections 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 --2- 172.21.15.76:0/762926642 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0f56f0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 -- 172.21.15.76:0/762926642 >> 172.21.15.76:0/762926642 conn(0x7fb33c0f0a30 msgr2=0x7fb33c0f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 -- 172.21.15.76:0/762926642 shutdown_connections 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.200+0000 7fb341224700 1 -- 172.21.15.76:0/762926642 wait complete. 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb341224700 1 Processor -- start 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb341224700 1 -- start start 2023-12-26T22:04:14.202 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb341224700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0fd2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb341224700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb33c0fb730 con 0x7fb33c0f52d0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb33bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0fd2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb33bfff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0fd2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:58254/0 (socket says 172.21.15.76:58254) 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb33bfff700 1 -- 172.21.15.76:0/1349423171 learned_addr learned my addr 172.21.15.76:0/1349423171 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.201+0000 7fb33bfff700 1 -- 172.21.15.76:0/1349423171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb32c005040 con 0x7fb33c0f52d0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb33bfff700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0fd2b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fb32c0089f0 tx=0x7fb32c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb32c009320 con 0x7fb33c0f52d0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fb32c013ec0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb33c0fb8d0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb32c014d30 con 0x7fb33c0f52d0 2023-12-26T22:04:14.203 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb33c0fbd60 con 0x7fb33c0f52d0 2023-12-26T22:04:14.204 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.202+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fb32c0263a0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.204 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.203+0000 7fb338ff9700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb324031090 0x7fb324033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:14.204 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.203+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fb32c0095f0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.204 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.203+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb33c0591e0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.204 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.203+0000 7fb33b7fe700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb324031090 0x7fb324033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:14.205 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.203+0000 7fb33b7fe700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb324031090 0x7fb324033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb3300096a0 tx=0x7fb330006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:14.207 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.208+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fb32c0266d0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.414 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.414+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb33c0fc3c0 con 0x7fb33c0f52d0 2023-12-26T22:04:14.415 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.415+0000 7fb338ff9700 1 -- 172.21.15.76:0/1349423171 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fb32c02b020 con 0x7fb33c0f52d0 2023-12-26T22:04:14.415 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:14.415 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb324031090 msgr2=0x7fb324033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb324031090 0x7fb324033550 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fb3300096a0 tx=0x7fb330006b40).stop 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 msgr2=0x7fb33c0fd2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0fd2b0 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7fb32c0089f0 tx=0x7fb32c00b040).stop 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 shutdown_connections 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fb324031090 0x7fb324033550 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 --2- 172.21.15.76:0/1349423171 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fb33c0f52d0 0x7fb33c0fd2b0 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 >> 172.21.15.76:0/1349423171 conn(0x7fb33c0f0a30 msgr2=0x7fb33c05ff00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 shutdown_connections 2023-12-26T22:04:14.417 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:14.417+0000 7fb341224700 1 -- 172.21.15.76:0/1349423171 wait complete. 2023-12-26T22:04:14.418 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:13.397062+0000 mgr.smithi059.gwkiyc (mgr.14156) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:04:13.586335+0000 mgr.smithi059.gwkiyc (mgr.14156) 233 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-761f1ff0-a439-11ee-95a5-87774f69a715-node-exporter.smithi059 2023-12-26T22:04:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: /bin/podman: stderr Error: no such container ceph-761f1ff0-a439-11ee-95a5-87774f69a715-node-exporter.smithi059 2023-12-26T22:04:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Deploy daemon node-exporter.smithi059 ... 2023-12-26T22:04:15.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Verifying port 9100 ... 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 1 from systemctl start ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: systemctl: stderr Job for ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059.service failed because a timeout was exceeded. 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: systemctl: stderr See "systemctl status ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059.service" and "journalctl -xe" for details. 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: main() 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: r = ctx.func(ctx) 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: return func(ctx) 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ports=daemon_ports) 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2023-12-26T22:04:15.295 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: RuntimeError: Failed command: systemctl start ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1347, in _remote_connection 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: yield (conn, connr) 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1244, in _run_cephadm 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: code, '\n'.join(err))) 2023-12-26T22:04:15.296 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-761f1ff0-a439-11ee-95a5-87774f69a715-node-exporter.smithi059 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: /bin/podman: stderr Error: no such container ceph-761f1ff0-a439-11ee-95a5-87774f69a715-node-exporter.smithi059 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Deploy daemon node-exporter.smithi059 ... 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Verifying port 9100 ... 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 1 from systemctl start ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: systemctl: stderr Job for ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059.service failed because a timeout was exceeded. 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: systemctl: stderr See "systemctl status ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059.service" and "journalctl -xe" for details. 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: main() 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: r = ctx.func(ctx) 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: return func(ctx) 2023-12-26T22:04:15.297 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ports=daemon_ports) 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: RuntimeError: Failed command: systemctl start ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:04:13.586557+0000 mgr.smithi059.gwkiyc (mgr.14156) 234 : cephadm [ERR] Failed while placing node-exporter.smithi059 on smithi059: cephadm exited with an error code: 1, stderr:Non-zero exit code 125 from /bin/podman container inspect --format {{.State.Status}} ceph-761f1ff0-a439-11ee-95a5-87774f69a715-node-exporter.smithi059 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: /bin/podman: stderr Error: no such container ceph-761f1ff0-a439-11ee-95a5-87774f69a715-node-exporter.smithi059 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Deploy daemon node-exporter.smithi059 ... 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Verifying port 9100 ... 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Non-zero exit code 1 from systemctl start ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059 2023-12-26T22:04:15.298 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: systemctl: stderr Job for ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059.service failed because a timeout was exceeded. 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: systemctl: stderr See "systemctl status ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059.service" and "journalctl -xe" for details. 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: Traceback (most recent call last): 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8230, in 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: main() 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 8218, in main 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: r = ctx.func(ctx) 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1759, in _default_image 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: return func(ctx) 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 4348, in command_deploy 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: ports=daemon_ports) 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2632, in deploy_daemon 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: c, osd_fsid=osd_fsid, ports=ports) 2023-12-26T22:04:15.299 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 2821, in deploy_daemon_units 2023-12-26T22:04:15.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-12-26T22:04:15.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: File "/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/cephadm.d4237e4639c108308fe13147b1c08af93c3d5724d9ff21ae797eb4b78fea3931", line 1453, in call_throws 2023-12-26T22:04:15.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-12-26T22:04:15.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: RuntimeError: Failed command: systemctl start ceph-761f1ff0-a439-11ee-95a5-87774f69a715@node-exporter.smithi059 2023-12-26T22:04:15.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cephadm 2023-12-26T22:04:13.592711+0000 mgr.smithi059.gwkiyc (mgr.14156) 235 : cephadm [INF] Deploying daemon prometheus.smithi059 on smithi059 2023-12-26T22:04:15.300 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:14.416762+0000 mon.smithi059 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.76:0/1349423171' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:15.810 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:15.810 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:16.793 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:15.397491+0000 mgr.smithi059.gwkiyc (mgr.14156) 236 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:16.794 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:15.480458+0000 mon.smithi059 (mon.0) 285 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' 2023-12-26T22:04:16.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.976+0000 7ff7630c1700 1 -- 172.21.15.76:0/2971406719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 msgr2=0x7ff75c0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.976+0000 7ff7630c1700 1 --2- 172.21.15.76:0/2971406719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c0f58a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff750009fb0 tx=0x7ff750009b30).stop 2023-12-26T22:04:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.976+0000 7ff7630c1700 1 -- 172.21.15.76:0/2971406719 shutdown_connections 2023-12-26T22:04:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.976+0000 7ff7630c1700 1 --2- 172.21.15.76:0/2971406719 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c0f58a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.976+0000 7ff7630c1700 1 -- 172.21.15.76:0/2971406719 >> 172.21.15.76:0/2971406719 conn(0x7ff75c0f0be0 msgr2=0x7ff75c0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.976+0000 7ff7630c1700 1 -- 172.21.15.76:0/2971406719 shutdown_connections 2023-12-26T22:04:16.977 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.977+0000 7ff7630c1700 1 -- 172.21.15.76:0/2971406719 wait complete. 2023-12-26T22:04:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.977+0000 7ff7630c1700 1 Processor -- start 2023-12-26T22:04:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.977+0000 7ff7630c1700 1 -- start start 2023-12-26T22:04:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff7630c1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c1094b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff7630c1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff75c060360 con 0x7ff75c0f5480 2023-12-26T22:04:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff760e5d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c1094b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:16.979 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff760e5d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c1094b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47210/0 (socket says 172.21.15.76:47210) 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff760e5d700 1 -- 172.21.15.76:0/3819340147 learned_addr learned my addr 172.21.15.76:0/3819340147 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff760e5d700 1 -- 172.21.15.76:0/3819340147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff750005040 con 0x7ff75c0f5480 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.978+0000 7ff760e5d700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c1094b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff750000c00 tx=0x7ff75000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.979+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff7500095f0 con 0x7ff75c0f5480 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.979+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff75c060560 con 0x7ff75c0f5480 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.979+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff750007e50 con 0x7ff75c0f5480 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.979+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff750014e70 con 0x7ff75c0f5480 2023-12-26T22:04:16.980 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.979+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff75c109be0 con 0x7ff75c0f5480 2023-12-26T22:04:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.980+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff750025070 con 0x7ff75c0f5480 2023-12-26T22:04:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.980+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff75c045bb0 con 0x7ff75c0f5480 2023-12-26T22:04:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.980+0000 7ff7597fa700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff74c0310e0 0x7ff74c0335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.980+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff750041cd0 con 0x7ff75c0f5480 2023-12-26T22:04:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.980+0000 7ff75bfff700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff74c0310e0 0x7ff74c0335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:16.981 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.980+0000 7ff75bfff700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff74c0310e0 0x7ff74c0335a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff7480096a0 tx=0x7ff748006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:16.985 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:16.985+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff7500079c0 con 0x7ff75c0f5480 2023-12-26T22:04:17.193 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.193+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff75c0faf70 con 0x7ff75c0f5480 2023-12-26T22:04:17.194 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.194+0000 7ff7597fa700 1 -- 172.21.15.76:0/3819340147 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff75c0faf70 con 0x7ff75c0f5480 2023-12-26T22:04:17.194 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:17.194 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:17.195 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff74c0310e0 msgr2=0x7ff74c0335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:17.195 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff74c0310e0 0x7ff74c0335a0 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ff7480096a0 tx=0x7ff748006b40).stop 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 msgr2=0x7ff75c1094b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c1094b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7ff750000c00 tx=0x7ff75000b040).stop 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 shutdown_connections 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff74c0310e0 0x7ff74c0335a0 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 --2- 172.21.15.76:0/3819340147 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff75c0f5480 0x7ff75c1094b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 >> 172.21.15.76:0/3819340147 conn(0x7ff75c0f0be0 msgr2=0x7ff75c0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 shutdown_connections 2023-12-26T22:04:17.196 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:17.195+0000 7ff7630c1700 1 -- 172.21.15.76:0/3819340147 wait complete. 2023-12-26T22:04:17.197 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:18.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:17 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:17.195499+0000 mon.smithi059 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.76:0/3819340147' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:18.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:18.572 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:19.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:17.397848+0000 mgr.smithi059.gwkiyc (mgr.14156) 237 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 -- 172.21.15.76:0/1326885550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 msgr2=0x7f06f40f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 --2- 172.21.15.76:0/1326885550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f40f58a0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f06e4009fb0 tx=0x7f06e4009b30).stop 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 -- 172.21.15.76:0/1326885550 shutdown_connections 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 --2- 172.21.15.76:0/1326885550 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f40f58a0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 -- 172.21.15.76:0/1326885550 >> 172.21.15.76:0/1326885550 conn(0x7f06f40f0be0 msgr2=0x7f06f40f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 -- 172.21.15.76:0/1326885550 shutdown_connections 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.790+0000 7f06fb34f700 1 -- 172.21.15.76:0/1326885550 wait complete. 2023-12-26T22:04:19.792 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06fb34f700 1 Processor -- start 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06fb34f700 1 -- start start 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06fb34f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f41094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06fb34f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06f4060320 con 0x7f06f40f5480 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06f90eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f41094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06f90eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f41094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47228/0 (socket says 172.21.15.76:47228) 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06f90eb700 1 -- 172.21.15.76:0/455320935 learned_addr learned my addr 172.21.15.76:0/455320935 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.791+0000 7f06f90eb700 1 -- 172.21.15.76:0/455320935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06e4005040 con 0x7f06f40f5480 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.792+0000 7f06f90eb700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f41094f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f06e40089f0 tx=0x7f06e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.792+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e4009320 con 0x7f06f40f5480 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.792+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f06e4013ec0 con 0x7f06f40f5480 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.792+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06f4060520 con 0x7f06f40f5480 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.792+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06e4014d30 con 0x7f06f40f5480 2023-12-26T22:04:19.793 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.792+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06f4109ba0 con 0x7f06f40f5480 2023-12-26T22:04:19.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.793+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f06e40095f0 con 0x7f06f40f5480 2023-12-26T22:04:19.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.793+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06f4045bb0 con 0x7f06f40f5480 2023-12-26T22:04:19.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.793+0000 7f06e9ffb700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f06e0031090 0x7f06e0033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:19.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.793+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f06e40424a0 con 0x7f06f40f5480 2023-12-26T22:04:19.794 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.793+0000 7f06f88ea700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f06e0031090 0x7f06e0033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:19.795 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.794+0000 7f06f88ea700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f06e0031090 0x7f06e0033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f06f00096a0 tx=0x7f06f0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:19.798 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:19.799+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f06e401d7f0 con 0x7f06f40f5480 2023-12-26T22:04:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.002+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06f40faf70 con 0x7f06f40f5480 2023-12-26T22:04:20.002 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.003+0000 7f06e9ffb700 1 -- 172.21.15.76:0/455320935 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f06e401d7f0 con 0x7f06f40f5480 2023-12-26T22:04:20.003 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:20.003 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:20.004 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f06e0031090 msgr2=0x7f06e0033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f06e0031090 0x7f06e0033550 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f06f00096a0 tx=0x7f06f0006b40).stop 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 msgr2=0x7f06f41094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f41094f0 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f06e40089f0 tx=0x7f06e400b040).stop 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 shutdown_connections 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f06e0031090 0x7f06e0033550 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 --2- 172.21.15.76:0/455320935 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f06f40f5480 0x7f06f41094f0 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.004+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 >> 172.21.15.76:0/455320935 conn(0x7f06f40f0be0 msgr2=0x7f06f40f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.005+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 shutdown_connections 2023-12-26T22:04:20.005 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:20.005+0000 7f06fb34f700 1 -- 172.21.15.76:0/455320935 wait complete. 2023-12-26T22:04:20.006 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:19.398269+0000 mgr.smithi059.gwkiyc (mgr.14156) 238 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:21.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:20.004309+0000 mon.smithi059 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.76:0/455320935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:21.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:21.393 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/1718333758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 msgr2=0x7f0e780f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 --2- 172.21.15.76:0/1718333758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e780f56f0 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f0e6c009fb0 tx=0x7f0e6c009b30).stop 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/1718333758 shutdown_connections 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 --2- 172.21.15.76:0/1718333758 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e780f56f0 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/1718333758 >> 172.21.15.76:0/1718333758 conn(0x7f0e780f0a30 msgr2=0x7f0e780f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/1718333758 shutdown_connections 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/1718333758 wait complete. 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 Processor -- start 2023-12-26T22:04:22.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- start start 2023-12-26T22:04:22.625 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e78109330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.625+0000 7f0e7d8ed700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0e78060300 con 0x7f0e780f52d0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e7c8eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e78109330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e7c8eb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e78109330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47234/0 (socket says 172.21.15.76:47234) 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e7c8eb700 1 -- 172.21.15.76:0/587804384 learned_addr learned my addr 172.21.15.76:0/587804384 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e7c8eb700 1 -- 172.21.15.76:0/587804384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0e6c005040 con 0x7f0e780f52d0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e7c8eb700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e78109330 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f0e6c0089f0 tx=0x7f0e6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e6c009320 con 0x7f0e780f52d0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0e78060500 con 0x7f0e780f52d0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f0e6c013ec0 con 0x7f0e780f52d0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.626+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0e6c014d30 con 0x7f0e780f52d0 2023-12-26T22:04:22.626 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.627+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0e781099e0 con 0x7f0e780f52d0 2023-12-26T22:04:22.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.627+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f0e6c014420 con 0x7f0e780f52d0 2023-12-26T22:04:22.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.627+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0e780591e0 con 0x7f0e780f52d0 2023-12-26T22:04:22.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.627+0000 7f0e757fa700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0e68031130 0x7f0e680335f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:22.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.627+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f0e6c042490 con 0x7f0e780f52d0 2023-12-26T22:04:22.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.628+0000 7f0e77fff700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0e68031130 0x7f0e680335f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:22.627 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.628+0000 7f0e77fff700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0e68031130 0x7f0e680335f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0e640096a0 tx=0x7f0e64006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:22.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.632+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f0e6c0268c0 con 0x7f0e780f52d0 2023-12-26T22:04:22.837 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.838+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0e78045bb0 con 0x7f0e780f52d0 2023-12-26T22:04:22.837 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.839+0000 7f0e757fa700 1 -- 172.21.15.76:0/587804384 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f0e6c01d380 con 0x7f0e780f52d0 2023-12-26T22:04:22.838 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:22.838 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:22.839 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0e68031130 msgr2=0x7f0e680335f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:22.839 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0e68031130 0x7f0e680335f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f0e640096a0 tx=0x7f0e64006b40).stop 2023-12-26T22:04:22.839 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 msgr2=0x7f0e78109330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:22.839 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e78109330 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f0e6c0089f0 tx=0x7f0e6c00b040).stop 2023-12-26T22:04:22.839 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 shutdown_connections 2023-12-26T22:04:22.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f0e68031130 0x7f0e680335f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:22.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 --2- 172.21.15.76:0/587804384 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f0e780f52d0 0x7f0e78109330 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:22.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 >> 172.21.15.76:0/587804384 conn(0x7f0e780f0a30 msgr2=0x7f0e780f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:22.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 shutdown_connections 2023-12-26T22:04:22.840 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:22.840+0000 7f0e7d8ed700 1 -- 172.21.15.76:0/587804384 wait complete. 2023-12-26T22:04:22.840 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:23.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:21.398607+0000 mgr.smithi059.gwkiyc (mgr.14156) 239 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:24.215 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:24.216 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:24.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:23 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:22.839368+0000 mon.smithi059 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.76:0/587804384' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:25.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:23.398987+0000 mgr.smithi059.gwkiyc (mgr.14156) 240 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:25.385 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 -- 172.21.15.76:0/3611304740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 msgr2=0x7fca400f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 --2- 172.21.15.76:0/3611304740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca400f9190 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fca30009fb0 tx=0x7fca30009b30).stop 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 -- 172.21.15.76:0/3611304740 shutdown_connections 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 --2- 172.21.15.76:0/3611304740 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca400f9190 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 -- 172.21.15.76:0/3611304740 >> 172.21.15.76:0/3611304740 conn(0x7fca400f0c10 msgr2=0x7fca400f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 -- 172.21.15.76:0/3611304740 shutdown_connections 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.385+0000 7fca462ef700 1 -- 172.21.15.76:0/3611304740 wait complete. 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca462ef700 1 Processor -- start 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca462ef700 1 -- start start 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca462ef700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca40106780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca462ef700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca40106c70 con 0x7fca40060360 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca3ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca40106780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca3ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca40106780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41040/0 (socket says 172.21.15.76:41040) 2023-12-26T22:04:25.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.386+0000 7fca3ffff700 1 -- 172.21.15.76:0/2744636292 learned_addr learned my addr 172.21.15.76:0/2744636292 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:25.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca3ffff700 1 -- 172.21.15.76:0/2744636292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca30005040 con 0x7fca40060360 2023-12-26T22:04:25.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca3ffff700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca40106780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fca30009f80 tx=0x7fca3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:25.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca30009320 con 0x7fca40060360 2023-12-26T22:04:25.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fca30013680 con 0x7fca40060360 2023-12-26T22:04:25.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca40106e70 con 0x7fca40060360 2023-12-26T22:04:25.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fca3001eda0 con 0x7fca40060360 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.387+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca40107300 con 0x7fca40060360 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.388+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fca3001e420 con 0x7fca40060360 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.388+0000 7fca3cff9700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fca28031090 0x7fca28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.388+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca40045bb0 con 0x7fca40060360 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.388+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fca30042440 con 0x7fca40060360 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.388+0000 7fca3f7fe700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fca28031090 0x7fca28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:25.388 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.388+0000 7fca3f7fe700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fca28031090 0x7fca28033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fca340096a0 tx=0x7fca34006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:25.391 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.393+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fca3001d380 con 0x7fca40060360 2023-12-26T22:04:25.599 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.600+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fca400fc090 con 0x7fca40060360 2023-12-26T22:04:25.600 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.601+0000 7fca3cff9700 1 -- 172.21.15.76:0/2744636292 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fca30004490 con 0x7fca40060360 2023-12-26T22:04:25.600 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:25.601 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fca28031090 msgr2=0x7fca28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fca28031090 0x7fca28033550 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fca340096a0 tx=0x7fca34006b40).stop 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 msgr2=0x7fca40106780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca40106780 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7fca30009f80 tx=0x7fca3000b040).stop 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 shutdown_connections 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fca28031090 0x7fca28033550 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 --2- 172.21.15.76:0/2744636292 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fca40060360 0x7fca40106780 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 >> 172.21.15.76:0/2744636292 conn(0x7fca400f0c10 msgr2=0x7fca400f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 shutdown_connections 2023-12-26T22:04:25.602 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:25.603+0000 7fca462ef700 1 -- 172.21.15.76:0/2744636292 wait complete. 2023-12-26T22:04:25.603 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:26.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:25 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:25.602015+0000 mon.smithi059 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.76:0/2744636292' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:26.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:26.972 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:27.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:25.399322+0000 mgr.smithi059.gwkiyc (mgr.14156) 241 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:28.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 -- 172.21.15.76:0/561259662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 msgr2=0x7feca00f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:28.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 --2- 172.21.15.76:0/561259662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca00f56f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fec90009fb0 tx=0x7fec90009b30).stop 2023-12-26T22:04:28.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 -- 172.21.15.76:0/561259662 shutdown_connections 2023-12-26T22:04:28.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 --2- 172.21.15.76:0/561259662 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca00f56f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:28.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 -- 172.21.15.76:0/561259662 >> 172.21.15.76:0/561259662 conn(0x7feca00f0a30 msgr2=0x7feca00f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:28.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 -- 172.21.15.76:0/561259662 shutdown_connections 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 -- 172.21.15.76:0/561259662 wait complete. 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.157+0000 7feca48a6700 1 Processor -- start 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7feca48a6700 1 -- start start 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7feca48a6700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca0106640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7feca48a6700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feca0106b30 con 0x7feca00f52d0 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7fec9f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca0106640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7fec9f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca0106640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41066/0 (socket says 172.21.15.76:41066) 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7fec9f7fe700 1 -- 172.21.15.76:0/1295188820 learned_addr learned my addr 172.21.15.76:0/1295188820 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7fec9f7fe700 1 -- 172.21.15.76:0/1295188820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec90005040 con 0x7feca00f52d0 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.158+0000 7fec9f7fe700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca0106640 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fec900089f0 tx=0x7fec9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.159+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec90009320 con 0x7feca00f52d0 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.159+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feca0106d30 con 0x7feca00f52d0 2023-12-26T22:04:28.160 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.159+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fec90013ec0 con 0x7feca00f52d0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.159+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fec90014d30 con 0x7feca00f52d0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.159+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feca01071c0 con 0x7feca00f52d0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.159+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fec900263a0 con 0x7feca00f52d0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.160+0000 7fec87fff700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec88031090 0x7fec88033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.160+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feca0045bb0 con 0x7feca00f52d0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.160+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fec90042490 con 0x7feca00f52d0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.160+0000 7fec9effd700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec88031090 0x7fec88033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:28.161 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.160+0000 7fec9effd700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec88031090 0x7fec88033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fec940096a0 tx=0x7fec94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:28.163 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.165+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fec9001db70 con 0x7feca00f52d0 2023-12-26T22:04:28.371 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.373+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feca0107fc0 con 0x7feca00f52d0 2023-12-26T22:04:28.372 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.373+0000 7fec87fff700 1 -- 172.21.15.76:0/1295188820 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fec9002b020 con 0x7feca00f52d0 2023-12-26T22:04:28.372 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:28.373 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec88031090 msgr2=0x7fec88033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec88031090 0x7fec88033550 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fec940096a0 tx=0x7fec94006b40).stop 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 msgr2=0x7feca0106640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca0106640 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fec900089f0 tx=0x7fec9000b040).stop 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 shutdown_connections 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fec88031090 0x7fec88033550 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:28.374 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 --2- 172.21.15.76:0/1295188820 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7feca00f52d0 0x7feca0106640 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:28.375 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 >> 172.21.15.76:0/1295188820 conn(0x7feca00f0a30 msgr2=0x7feca00f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:28.375 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 shutdown_connections 2023-12-26T22:04:28.375 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:28.375+0000 7feca48a6700 1 -- 172.21.15.76:0/1295188820 wait complete. 2023-12-26T22:04:28.375 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:27.399672+0000 mgr.smithi059.gwkiyc (mgr.14156) 242 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:29.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:28.374080+0000 mon.smithi059 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.76:0/1295188820' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:29.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:29.757 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:30.956 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 -- 172.21.15.76:0/238590656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 msgr2=0x7f54e80f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 --2- 172.21.15.76:0/238590656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80f58a0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f54d8009fb0 tx=0x7f54d8009b30).stop 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 -- 172.21.15.76:0/238590656 shutdown_connections 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 --2- 172.21.15.76:0/238590656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80f58a0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 -- 172.21.15.76:0/238590656 >> 172.21.15.76:0/238590656 conn(0x7f54e80f0be0 msgr2=0x7f54e80f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 -- 172.21.15.76:0/238590656 shutdown_connections 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 -- 172.21.15.76:0/238590656 wait complete. 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.956+0000 7f54ee679700 1 Processor -- start 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54ee679700 1 -- start start 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54ee679700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54ee679700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54e80fb8e0 con 0x7f54e80f5480 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54e7fff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41086/0 (socket says 172.21.15.76:41086) 2023-12-26T22:04:30.957 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54e7fff700 1 -- 172.21.15.76:0/452551851 learned_addr learned my addr 172.21.15.76:0/452551851 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54e7fff700 1 -- 172.21.15.76:0/452551851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d8005040 con 0x7f54e80f5480 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.957+0000 7f54e7fff700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80fd460 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f54d80089f0 tx=0x7f54d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.958+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54d8009320 con 0x7f54e80f5480 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.958+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54e80fba80 con 0x7f54e80f5480 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.958+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f54d8013ec0 con 0x7f54e80f5480 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.958+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f54d8014d30 con 0x7f54e80f5480 2023-12-26T22:04:30.958 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.958+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54e80fbf10 con 0x7f54e80f5480 2023-12-26T22:04:30.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.958+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f54d80263a0 con 0x7f54e80f5480 2023-12-26T22:04:30.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.959+0000 7f54e4ff9700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54d40310e0 0x7f54d40335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:30.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.959+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f54d80424a0 con 0x7f54e80f5480 2023-12-26T22:04:30.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.959+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54e80591e0 con 0x7f54e80f5480 2023-12-26T22:04:30.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.959+0000 7f54e77fe700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54d40310e0 0x7f54d40335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:30.959 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.959+0000 7f54e77fe700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54d40310e0 0x7f54d40335a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f54d00096a0 tx=0x7f54d0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:30.962 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:30.963+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f54d80266d0 con 0x7f54e80f5480 2023-12-26T22:04:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.170+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54e80fc570 con 0x7f54e80f5480 2023-12-26T22:04:31.170 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.171+0000 7f54e4ff9700 1 -- 172.21.15.76:0/452551851 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f54d8012330 con 0x7f54e80f5480 2023-12-26T22:04:31.170 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:31.170 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54d40310e0 msgr2=0x7f54d40335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54d40310e0 0x7f54d40335a0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f54d00096a0 tx=0x7f54d0006b40).stop 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 msgr2=0x7f54e80fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80fd460 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f54d80089f0 tx=0x7f54d800b040).stop 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 shutdown_connections 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f54d40310e0 0x7f54d40335a0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 --2- 172.21.15.76:0/452551851 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f54e80f5480 0x7f54e80fd460 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 >> 172.21.15.76:0/452551851 conn(0x7f54e80f0be0 msgr2=0x7f54e805ff40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 shutdown_connections 2023-12-26T22:04:31.172 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:31.173+0000 7f54ee679700 1 -- 172.21.15.76:0/452551851 wait complete. 2023-12-26T22:04:31.173 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:31.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:29.400010+0000 mgr.smithi059.gwkiyc (mgr.14156) 243 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:32.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:31.171914+0000 mon.smithi059 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.76:0/452551851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:32.549 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:32.549 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:33.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:32 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:31.400372+0000 mgr.smithi059.gwkiyc (mgr.14156) 244 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.772+0000 7fbbaaf81700 1 -- 172.21.15.76:0/3600313175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 msgr2=0x7fbba40f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.772+0000 7fbbaaf81700 1 --2- 172.21.15.76:0/3600313175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba40f56f0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fbba0009fb0 tx=0x7fbba0009b30).stop 2023-12-26T22:04:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 -- 172.21.15.76:0/3600313175 shutdown_connections 2023-12-26T22:04:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 --2- 172.21.15.76:0/3600313175 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba40f56f0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:33.772 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 -- 172.21.15.76:0/3600313175 >> 172.21.15.76:0/3600313175 conn(0x7fbba40f0a30 msgr2=0x7fbba40f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:33.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 -- 172.21.15.76:0/3600313175 shutdown_connections 2023-12-26T22:04:33.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 -- 172.21.15.76:0/3600313175 wait complete. 2023-12-26T22:04:33.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 Processor -- start 2023-12-26T22:04:33.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 -- start start 2023-12-26T22:04:33.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba4106600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:33.773 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.773+0000 7fbbaaf81700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba4106af0 con 0x7fbba40f52d0 2023-12-26T22:04:33.774 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbba9f7f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba4106600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:33.774 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbba9f7f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba4106600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:41114/0 (socket says 172.21.15.76:41114) 2023-12-26T22:04:33.774 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbba9f7f700 1 -- 172.21.15.76:0/92008576 learned_addr learned my addr 172.21.15.76:0/92008576 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:33.774 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbba9f7f700 1 -- 172.21.15.76:0/92008576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba0005040 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbba9f7f700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba4106600 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fbba00089f0 tx=0x7fbba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba0009320 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fbba0013ec0 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba4106cf0 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba0014d30 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.774+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba4107180 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.775+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fbba00095f0 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.775+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba4045bb0 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.775+0000 7fbb9a7fc700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb90031090 0x7fbb90033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.775+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fbba00423e0 con 0x7fbba40f52d0 2023-12-26T22:04:33.775 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.776+0000 7fbba977e700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb90031090 0x7fbb90033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:33.776 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.776+0000 7fbba977e700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb90031090 0x7fbb90033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbb940096a0 tx=0x7fbb94006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:33.779 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.781+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fbba001d7f0 con 0x7fbba40f52d0 2023-12-26T22:04:33.987 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.988+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbba4108030 con 0x7fbba40f52d0 2023-12-26T22:04:33.988 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.989+0000 7fbb9a7fc700 1 -- 172.21.15.76:0/92008576 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fbba001d7f0 con 0x7fbba40f52d0 2023-12-26T22:04:33.988 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:33.988 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:33.990 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb90031090 msgr2=0x7fbb90033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:33.990 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb90031090 0x7fbb90033550 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fbb940096a0 tx=0x7fbb94006b40).stop 2023-12-26T22:04:33.990 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 msgr2=0x7fbba4106600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:33.990 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba4106600 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fbba00089f0 tx=0x7fbba000b040).stop 2023-12-26T22:04:33.990 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 shutdown_connections 2023-12-26T22:04:33.990 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fbb90031090 0x7fbb90033550 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:33.991 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 --2- 172.21.15.76:0/92008576 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fbba40f52d0 0x7fbba4106600 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:33.991 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 >> 172.21.15.76:0/92008576 conn(0x7fbba40f0a30 msgr2=0x7fbba405fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:33.991 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 shutdown_connections 2023-12-26T22:04:33.991 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:33.991+0000 7fbbaaf81700 1 -- 172.21.15.76:0/92008576 wait complete. 2023-12-26T22:04:33.991 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:35.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:33.400586+0000 mgr.smithi059.gwkiyc (mgr.14156) 245 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:35.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:34 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:33.989670+0000 mon.smithi059 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.76:0/92008576' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:35.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:35.367 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 -- 172.21.15.76:0/23315878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 msgr2=0x7f47600f56f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 --2- 172.21.15.76:0/23315878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47600f56f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f4750009fb0 tx=0x7f4750009b30).stop 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 -- 172.21.15.76:0/23315878 shutdown_connections 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 --2- 172.21.15.76:0/23315878 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47600f56f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 -- 172.21.15.76:0/23315878 >> 172.21.15.76:0/23315878 conn(0x7f47600f0a30 msgr2=0x7f47600f2e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 -- 172.21.15.76:0/23315878 shutdown_connections 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 -- 172.21.15.76:0/23315878 wait complete. 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.550+0000 7f476485d700 1 Processor -- start 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f476485d700 1 -- start start 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f476485d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47601092d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f476485d700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4760060360 con 0x7f47600f52d0 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f475ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47601092d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:36.551 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f475ed9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47601092d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60672/0 (socket says 172.21.15.76:60672) 2023-12-26T22:04:36.552 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f475ed9d700 1 -- 172.21.15.76:0/4192740355 learned_addr learned my addr 172.21.15.76:0/4192740355 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f475ed9d700 1 -- 172.21.15.76:0/4192740355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4750005040 con 0x7f47600f52d0 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.551+0000 7f475ed9d700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47601092d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f47500089f0 tx=0x7f475000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.552+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4750009320 con 0x7f47600f52d0 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.552+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4760060560 con 0x7f47600f52d0 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.552+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f4750013ec0 con 0x7f47600f52d0 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.552+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4750014d30 con 0x7f47600f52d0 2023-12-26T22:04:36.553 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.552+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4760109a00 con 0x7f47600f52d0 2023-12-26T22:04:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.552+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f4750014420 con 0x7f47600f52d0 2023-12-26T22:04:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.553+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4760045bb0 con 0x7f47600f52d0 2023-12-26T22:04:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.553+0000 7f47477fe700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4748031090 0x7f4748033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.553+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f47500424c0 con 0x7f47600f52d0 2023-12-26T22:04:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.553+0000 7f475e59c700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4748031090 0x7f4748033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:36.554 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.553+0000 7f475e59c700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4748031090 0x7f4748033550 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f47540096a0 tx=0x7f4754006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:36.557 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.557+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f475001db60 con 0x7f47600f52d0 2023-12-26T22:04:36.765 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.765+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f47600fadc0 con 0x7f47600f52d0 2023-12-26T22:04:36.765 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.766+0000 7f47477fe700 1 -- 172.21.15.76:0/4192740355 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f475001d380 con 0x7f47600f52d0 2023-12-26T22:04:36.766 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:36.766 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4748031090 msgr2=0x7f4748033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4748031090 0x7f4748033550 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f47540096a0 tx=0x7f4754006b40).stop 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 msgr2=0x7f47601092d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47601092d0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f47500089f0 tx=0x7f475000b040).stop 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 shutdown_connections 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f4748031090 0x7f4748033550 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 --2- 172.21.15.76:0/4192740355 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f47600f52d0 0x7f47601092d0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:36.767 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 >> 172.21.15.76:0/4192740355 conn(0x7f47600f0a30 msgr2=0x7f47600f1550 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:36.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 shutdown_connections 2023-12-26T22:04:36.768 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:36.767+0000 7f476485d700 1 -- 172.21.15.76:0/4192740355 wait complete. 2023-12-26T22:04:36.768 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:35.400888+0000 mgr.smithi059.gwkiyc (mgr.14156) 246 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:37.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:36 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:36.767315+0000 mon.smithi059 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.76:0/4192740355' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:38.146 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:38.147 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:39.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:38 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:37.401247+0000 mgr.smithi059.gwkiyc (mgr.14156) 247 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:39.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/3912037981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 msgr2=0x7f2d080f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:39.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 --2- 172.21.15.76:0/3912037981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d080f6800 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f2d04009fb0 tx=0x7f2d04009b30).stop 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/3912037981 shutdown_connections 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 --2- 172.21.15.76:0/3912037981 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d080f6800 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/3912037981 >> 172.21.15.76:0/3912037981 conn(0x7f2d080f1b40 msgr2=0x7f2d080f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/3912037981 shutdown_connections 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/3912037981 wait complete. 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 Processor -- start 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.383+0000 7f2d0e4cd700 1 -- start start 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0e4cd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d0806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0e4cd700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08069f00 con 0x7f2d080f63e0 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0d4cb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d0806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0d4cb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d0806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60678/0 (socket says 172.21.15.76:60678) 2023-12-26T22:04:39.384 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0d4cb700 1 -- 172.21.15.76:0/1634350446 learned_addr learned my addr 172.21.15.76:0/1634350446 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0d4cb700 1 -- 172.21.15.76:0/1634350446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d04005040 con 0x7f2d080f63e0 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0d4cb700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d0806ba80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2d040089f0 tx=0x7f2d0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d04009320 con 0x7f2d080f63e0 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2d04013ec0 con 0x7f2d080f63e0 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d0806a0a0 con 0x7f2d080f63e0 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.384+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2d04014d30 con 0x7f2d080f63e0 2023-12-26T22:04:39.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.385+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0806a530 con 0x7f2d080f63e0 2023-12-26T22:04:39.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.385+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d080591e0 con 0x7f2d080f63e0 2023-12-26T22:04:39.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.385+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2d040095f0 con 0x7f2d080f63e0 2023-12-26T22:04:39.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.386+0000 7f2cfdffb700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cf4031090 0x7f2cf4033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:39.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.386+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2d04042420 con 0x7f2d080f63e0 2023-12-26T22:04:39.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.386+0000 7f2d0ccca700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cf4031090 0x7f2cf4033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:39.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.386+0000 7f2d0ccca700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cf4031090 0x7f2cf4033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f2cf80096a0 tx=0x7f2cf8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:39.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.391+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2d0401d7f0 con 0x7f2d080f63e0 2023-12-26T22:04:39.596 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.596+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2d08045bb0 con 0x7f2d080f63e0 2023-12-26T22:04:39.596 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.597+0000 7f2cfdffb700 1 -- 172.21.15.76:0/1634350446 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2d0400fe40 con 0x7f2d080f63e0 2023-12-26T22:04:39.597 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:39.597 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:39.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cf4031090 msgr2=0x7f2cf4033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:39.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cf4031090 0x7f2cf4033550 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f2cf80096a0 tx=0x7f2cf8006b40).stop 2023-12-26T22:04:39.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 msgr2=0x7f2d0806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:39.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d0806ba80 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f2d040089f0 tx=0x7f2d0400b040).stop 2023-12-26T22:04:39.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 shutdown_connections 2023-12-26T22:04:39.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2cf4031090 0x7f2cf4033550 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:39.599 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 --2- 172.21.15.76:0/1634350446 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2d080f63e0 0x7f2d0806ba80 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:39.599 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 >> 172.21.15.76:0/1634350446 conn(0x7f2d080f1b40 msgr2=0x7f2d080f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:39.599 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 shutdown_connections 2023-12-26T22:04:39.599 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:39.598+0000 7f2d0e4cd700 1 -- 172.21.15.76:0/1634350446 wait complete. 2023-12-26T22:04:39.599 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:40.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:39 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:39.598272+0000 mon.smithi059 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.76:0/1634350446' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:41.046 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:41.047 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:41.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:40 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:39.401559+0000 mgr.smithi059.gwkiyc (mgr.14156) 248 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 -- 172.21.15.76:0/2601815380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 msgr2=0x7f50500f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 --2- 172.21.15.76:0/2601815380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f9160 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f5044009fb0 tx=0x7f5044009b30).stop 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 -- 172.21.15.76:0/2601815380 shutdown_connections 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 --2- 172.21.15.76:0/2601815380 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f9160 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 -- 172.21.15.76:0/2601815380 >> 172.21.15.76:0/2601815380 conn(0x7f50500f0be0 msgr2=0x7f50500f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 -- 172.21.15.76:0/2601815380 shutdown_connections 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.281+0000 7f5057578700 1 -- 172.21.15.76:0/2601815380 wait complete. 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5057578700 1 Processor -- start 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5057578700 1 -- start start 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5057578700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5057578700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50500f5060 con 0x7f5050060360 2023-12-26T22:04:42.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5055314700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5055314700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60702/0 (socket says 172.21.15.76:60702) 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.282+0000 7f5055314700 1 -- 172.21.15.76:0/4216070644 learned_addr learned my addr 172.21.15.76:0/4216070644 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5055314700 1 -- 172.21.15.76:0/4216070644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5044005040 con 0x7f5050060360 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5055314700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f6be0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f5044009f80 tx=0x7f504400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5044009320 con 0x7f5050060360 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50500f5200 con 0x7f5050060360 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5044013680 con 0x7f5050060360 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f504401eda0 con 0x7f5050060360 2023-12-26T22:04:42.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.283+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50500f5690 con 0x7f5050060360 2023-12-26T22:04:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.284+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f504401e420 con 0x7f5050060360 2023-12-26T22:04:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.284+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50500591e0 con 0x7f5050060360 2023-12-26T22:04:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.284+0000 7f5041ffb700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f503c031090 0x7f503c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.284+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5044042450 con 0x7f5050060360 2023-12-26T22:04:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.284+0000 7f5054b13700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f503c031090 0x7f503c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:42.285 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.285+0000 7f5054b13700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f503c031090 0x7f503c033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f504c0096a0 tx=0x7f504c006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:42.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.289+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f504401d380 con 0x7f5050060360 2023-12-26T22:04:42.494 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.494+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5050045bb0 con 0x7f5050060360 2023-12-26T22:04:42.994 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.495+0000 7f5041ffb700 1 -- 172.21.15.76:0/4216070644 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f504401cb10 con 0x7f5050060360 2023-12-26T22:04:42.994 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:42.995 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:42.995 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f503c031090 msgr2=0x7f503c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f503c031090 0x7f503c033550 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f504c0096a0 tx=0x7f504c006b40).stop 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 msgr2=0x7f50500f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f6be0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f5044009f80 tx=0x7f504400b040).stop 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 shutdown_connections 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f503c031090 0x7f503c033550 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 --2- 172.21.15.76:0/4216070644 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f5050060360 0x7f50500f6be0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 >> 172.21.15.76:0/4216070644 conn(0x7f50500f0be0 msgr2=0x7f50500f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 shutdown_connections 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:42.496+0000 7f5057578700 1 -- 172.21.15.76:0/4216070644 wait complete. 2023-12-26T22:04:42.996 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:43.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:41.401927+0000 mgr.smithi059.gwkiyc (mgr.14156) 249 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:43.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:42 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:42.496339+0000 mon.smithi059 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.76:0/4216070644' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:43.997 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:43.997 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 -- 172.21.15.76:0/217454693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 msgr2=0x7fdcb00f6cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 --2- 172.21.15.76:0/217454693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb00f6cd0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fdca0009fb0 tx=0x7fdca0009b30).stop 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 -- 172.21.15.76:0/217454693 shutdown_connections 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 --2- 172.21.15.76:0/217454693 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb00f6cd0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 -- 172.21.15.76:0/217454693 >> 172.21.15.76:0/217454693 conn(0x7fdcb00f0190 msgr2=0x7fdcb00f25d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 -- 172.21.15.76:0/217454693 shutdown_connections 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 -- 172.21.15.76:0/217454693 wait complete. 2023-12-26T22:04:45.171 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.171+0000 7fdcb57a4700 1 Processor -- start 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcb57a4700 1 -- start start 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcb57a4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb0105510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcb57a4700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb0105a00 con 0x7fdcb00f68b0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcaffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb0105510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcaffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb0105510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:60726/0 (socket says 172.21.15.76:60726) 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcaffff700 1 -- 172.21.15.76:0/1789831212 learned_addr learned my addr 172.21.15.76:0/1789831212 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcaffff700 1 -- 172.21.15.76:0/1789831212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdca0005040 con 0x7fdcb00f68b0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.172+0000 7fdcaffff700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb0105510 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fdca00089f0 tx=0x7fdca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.173+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca0009320 con 0x7fdcb00f68b0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.173+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb0105c00 con 0x7fdcb00f68b0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.173+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fdca0013ec0 con 0x7fdcb00f68b0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.173+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdca0014d30 con 0x7fdcb00f68b0 2023-12-26T22:04:45.173 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.173+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb0106090 con 0x7fdcb00f68b0 2023-12-26T22:04:45.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.173+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fdca00095f0 con 0x7fdcb00f68b0 2023-12-26T22:04:45.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.174+0000 7fdcacff9700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc98031090 0x7fdc98033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:45.174 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.174+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb0045bb0 con 0x7fdcb00f68b0 2023-12-26T22:04:45.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.174+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fdca00423e0 con 0x7fdcb00f68b0 2023-12-26T22:04:45.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.174+0000 7fdcaf7fe700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc98031090 0x7fdc98033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:45.175 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.174+0000 7fdcaf7fe700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc98031090 0x7fdc98033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fdca40096a0 tx=0x7fdca4006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:45.178 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.179+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fdca001d7f0 con 0x7fdcb00f68b0 2023-12-26T22:04:45.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:44 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:43.402360+0000 mgr.smithi059.gwkiyc (mgr.14156) 250 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.387+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdcb0106f10 con 0x7fdcb00f68b0 2023-12-26T22:04:45.387 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.388+0000 7fdcacff9700 1 -- 172.21.15.76:0/1789831212 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fdca001d7f0 con 0x7fdcb00f68b0 2023-12-26T22:04:45.388 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:45.388 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:45.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc98031090 msgr2=0x7fdc98033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:45.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc98031090 0x7fdc98033550 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fdca40096a0 tx=0x7fdca4006b40).stop 2023-12-26T22:04:45.389 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 msgr2=0x7fdcb0105510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb0105510 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fdca00089f0 tx=0x7fdca000b040).stop 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 shutdown_connections 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fdc98031090 0x7fdc98033550 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 --2- 172.21.15.76:0/1789831212 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fdcb00f68b0 0x7fdcb0105510 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 >> 172.21.15.76:0/1789831212 conn(0x7fdcb00f0190 msgr2=0x7fdcb00f0cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 shutdown_connections 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:45.389+0000 7fdcb57a4700 1 -- 172.21.15.76:0/1789831212 wait complete. 2023-12-26T22:04:45.390 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:46.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:45.389243+0000 mon.smithi059 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.76:0/1789831212' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:45.473421+0000 mon.smithi059 (mon.0) 297 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/mirror_snapshot_schedule"}]: dispatch 2023-12-26T22:04:46.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:45 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:45.473657+0000 mon.smithi059 (mon.0) 298 : audit [INF] from='mgr.14156 172.21.15.59:0/1889274930' entity='mgr.smithi059.gwkiyc' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi059.gwkiyc/trash_purge_schedule"}]: dispatch 2023-12-26T22:04:46.776 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:46.777 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:47.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:46 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:45.402740+0000 mgr.smithi059.gwkiyc (mgr.14156) 251 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.968+0000 7f9979e35700 1 -- 172.21.15.76:0/2371546570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 msgr2=0x7f99740f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.968+0000 7f9979e35700 1 --2- 172.21.15.76:0/2371546570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f99740f6800 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f9968009fb0 tx=0x7f9968009b30).stop 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 -- 172.21.15.76:0/2371546570 shutdown_connections 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 --2- 172.21.15.76:0/2371546570 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f99740f6800 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 -- 172.21.15.76:0/2371546570 >> 172.21.15.76:0/2371546570 conn(0x7f99740f1b40 msgr2=0x7f99740f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 -- 172.21.15.76:0/2371546570 shutdown_connections 2023-12-26T22:04:47.969 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 -- 172.21.15.76:0/2371546570 wait complete. 2023-12-26T22:04:47.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 Processor -- start 2023-12-26T22:04:47.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 -- start start 2023-12-26T22:04:47.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f997406ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:47.970 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.969+0000 7f9979e35700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974069f00 con 0x7f99740f63e0 2023-12-26T22:04:47.971 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9978e33700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f997406ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9978e33700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f997406ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39284/0 (socket says 172.21.15.76:39284) 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9978e33700 1 -- 172.21.15.76:0/3466393547 learned_addr learned my addr 172.21.15.76:0/3466393547 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9978e33700 1 -- 172.21.15.76:0/3466393547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9968005040 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9978e33700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f997406ba80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f99680089f0 tx=0x7f996800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9968009320 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f997406a0a0 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f997406a530 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f9968013ec0 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.970+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9968014d30 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.971+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f99680095f0 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.971+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99740591e0 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.971+0000 7f99717fa700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9964031090 0x7f9964033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.971+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f9968042530 con 0x7f99740f63e0 2023-12-26T22:04:47.972 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.971+0000 7f9973fff700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9964031090 0x7f9964033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:47.973 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.972+0000 7f9973fff700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9964031090 0x7f9964033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f99600096a0 tx=0x7f9960006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:47.976 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:47.976+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f996801d7f0 con 0x7f99740f63e0 2023-12-26T22:04:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.182+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99740fbed0 con 0x7f99740f63e0 2023-12-26T22:04:48.182 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.183+0000 7f99717fa700 1 -- 172.21.15.76:0/3466393547 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f996801d7f0 con 0x7f99740f63e0 2023-12-26T22:04:48.183 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:48.183 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9964031090 msgr2=0x7f9964033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9964031090 0x7f9964033550 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f99600096a0 tx=0x7f9960006b40).stop 2023-12-26T22:04:48.184 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 msgr2=0x7f997406ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f997406ba80 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f99680089f0 tx=0x7f996800b040).stop 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 shutdown_connections 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9964031090 0x7f9964033550 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 --2- 172.21.15.76:0/3466393547 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f99740f63e0 0x7f997406ba80 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.184+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 >> 172.21.15.76:0/3466393547 conn(0x7f99740f1b40 msgr2=0x7f99740f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.185+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 shutdown_connections 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:48.185+0000 7f9979e35700 1 -- 172.21.15.76:0/3466393547 wait complete. 2023-12-26T22:04:48.185 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:49.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:47.403090+0000 mgr.smithi059.gwkiyc (mgr.14156) 252 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:49.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:48 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:48.184311+0000 mon.smithi059 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.76:0/3466393547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:49.567 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:49.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 -- 172.21.15.76:0/1284840159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 msgr2=0x7fe0780f5030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 --2- 172.21.15.76:0/1284840159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0780f5030 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fe06c009fb0 tx=0x7fe06c009b30).stop 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 -- 172.21.15.76:0/1284840159 shutdown_connections 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 --2- 172.21.15.76:0/1284840159 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0780f5030 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 -- 172.21.15.76:0/1284840159 >> 172.21.15.76:0/1284840159 conn(0x7fe0780f0390 msgr2=0x7fe0780f27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 -- 172.21.15.76:0/1284840159 shutdown_connections 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.880+0000 7fe07f07a700 1 -- 172.21.15.76:0/1284840159 wait complete. 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.881+0000 7fe07f07a700 1 Processor -- start 2023-12-26T22:04:50.881 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.881+0000 7fe07f07a700 1 -- start start 2023-12-26T22:04:50.882 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.881+0000 7fe07f07a700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0781067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:50.882 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.881+0000 7fe07f07a700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe078106ce0 con 0x7fe0780f4c10 2023-12-26T22:04:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.881+0000 7fe07ce16700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0781067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe07ce16700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0781067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39298/0 (socket says 172.21.15.76:39298) 2023-12-26T22:04:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe07ce16700 1 -- 172.21.15.76:0/3863542291 learned_addr learned my addr 172.21.15.76:0/3863542291 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe07ce16700 1 -- 172.21.15.76:0/3863542291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe06c005040 con 0x7fe0780f4c10 2023-12-26T22:04:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe07ce16700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0781067f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe06c0089f0 tx=0x7fe06c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:50.883 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c009320 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe078106ee0 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fe06c013ec0 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe06c014d30 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.882+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe078107370 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.883+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fe06c0263a0 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.883+0000 7fe0757fa700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0680310e0 0x7fe0680335a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.883+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fe06c0423c0 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.883+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0780591e0 con 0x7fe0780f4c10 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.883+0000 7fe077fff700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0680310e0 0x7fe0680335a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:50.884 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.884+0000 7fe077fff700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0680310e0 0x7fe0680335a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:50.888 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:50.888+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fe06c012920 con 0x7fe0780f4c10 2023-12-26T22:04:51.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.093+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe078045bb0 con 0x7fe0780f4c10 2023-12-26T22:04:51.093 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.094+0000 7fe0757fa700 1 -- 172.21.15.76:0/3863542291 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fe06c012330 con 0x7fe0780f4c10 2023-12-26T22:04:51.094 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:51.094 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0680310e0 msgr2=0x7fe0680335a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0680310e0 0x7fe0680335a0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe0640096a0 tx=0x7fe064006b40).stop 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 msgr2=0x7fe0781067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0781067f0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fe06c0089f0 tx=0x7fe06c00b040).stop 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 shutdown_connections 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fe0680310e0 0x7fe0680335a0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 --2- 172.21.15.76:0/3863542291 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fe0780f4c10 0x7fe0781067f0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:51.095 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 >> 172.21.15.76:0/3863542291 conn(0x7fe0780f0390 msgr2=0x7fe0780f1590 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:51.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 shutdown_connections 2023-12-26T22:04:51.096 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:51.095+0000 7fe07f07a700 1 -- 172.21.15.76:0/3863542291 wait complete. 2023-12-26T22:04:51.096 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:51.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:50 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:49.403499+0000 mgr.smithi059.gwkiyc (mgr.14156) 253 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:52.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:51 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:51.095318+0000 mon.smithi059 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.76:0/3863542291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:52.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:52.478 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:53.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:52 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:51.403888+0000 mgr.smithi059.gwkiyc (mgr.14156) 254 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:53.628 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 -- 172.21.15.76:0/3761231701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 msgr2=0x7ff2400f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 --2- 172.21.15.76:0/3761231701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f9160 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7ff234009fb0 tx=0x7ff234009b30).stop 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 -- 172.21.15.76:0/3761231701 shutdown_connections 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 --2- 172.21.15.76:0/3761231701 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f9160 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 -- 172.21.15.76:0/3761231701 >> 172.21.15.76:0/3761231701 conn(0x7ff2400f0be0 msgr2=0x7ff2400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 -- 172.21.15.76:0/3761231701 shutdown_connections 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.628+0000 7ff246918700 1 -- 172.21.15.76:0/3761231701 wait complete. 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff246918700 1 Processor -- start 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff246918700 1 -- start start 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff246918700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff246918700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2400f5060 con 0x7ff240060360 2023-12-26T22:04:53.629 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff23ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff23ffff700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:39306/0 (socket says 172.21.15.76:39306) 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff23ffff700 1 -- 172.21.15.76:0/2136833473 learned_addr learned my addr 172.21.15.76:0/2136833473 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.629+0000 7ff23ffff700 1 -- 172.21.15.76:0/2136833473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff234005040 con 0x7ff240060360 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.630+0000 7ff23ffff700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f6be0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ff234009f80 tx=0x7ff23400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.630+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff234009320 con 0x7ff240060360 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.630+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7ff234013680 con 0x7ff240060360 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.630+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2400f5200 con 0x7ff240060360 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.630+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff23401eda0 con 0x7ff240060360 2023-12-26T22:04:53.630 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.630+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2400f5690 con 0x7ff240060360 2023-12-26T22:04:53.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.631+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7ff23401e420 con 0x7ff240060360 2023-12-26T22:04:53.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.631+0000 7ff23cff9700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff228031090 0x7ff228033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:53.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.631+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2400591e0 con 0x7ff240060360 2023-12-26T22:04:53.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.631+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7ff234042390 con 0x7ff240060360 2023-12-26T22:04:53.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.631+0000 7ff23f7fe700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff228031090 0x7ff228033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:53.631 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.632+0000 7ff23f7fe700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff228031090 0x7ff228033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff2300096a0 tx=0x7ff230006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:53.635 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.636+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7ff23401e700 con 0x7ff240060360 2023-12-26T22:04:53.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.845+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2400fc060 con 0x7ff240060360 2023-12-26T22:04:53.844 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.846+0000 7ff23cff9700 1 -- 172.21.15.76:0/2136833473 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7ff23401cb10 con 0x7ff240060360 2023-12-26T22:04:53.845 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:53.845 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:53.846 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff228031090 msgr2=0x7ff228033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:53.846 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff228031090 0x7ff228033550 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7ff2300096a0 tx=0x7ff230006b40).stop 2023-12-26T22:04:53.846 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 msgr2=0x7ff2400f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:53.846 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f6be0 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7ff234009f80 tx=0x7ff23400b040).stop 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 shutdown_connections 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7ff228031090 0x7ff228033550 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 --2- 172.21.15.76:0/2136833473 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7ff240060360 0x7ff2400f6be0 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 >> 172.21.15.76:0/2136833473 conn(0x7ff2400f0be0 msgr2=0x7ff2400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 shutdown_connections 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:53.847+0000 7ff246918700 1 -- 172.21.15.76:0/2136833473 wait complete. 2023-12-26T22:04:53.847 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:54.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:53 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:53.846521+0000 mon.smithi059 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.76:0/2136833473' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:55.227 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:55.228 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:55.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:54 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:53.404215+0000 mgr.smithi059.gwkiyc (mgr.14156) 255 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:56.381 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 -- 172.21.15.76:0/952790396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 msgr2=0x7f20980f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:56.381 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 --2- 172.21.15.76:0/952790396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f20980f69b0 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f2094004760 tx=0x7f2094009ad0).stop 2023-12-26T22:04:56.381 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 -- 172.21.15.76:0/952790396 shutdown_connections 2023-12-26T22:04:56.381 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 --2- 172.21.15.76:0/952790396 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f20980f69b0 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:56.381 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 -- 172.21.15.76:0/952790396 >> 172.21.15.76:0/952790396 conn(0x7f20980f1cd0 msgr2=0x7f20980f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:56.381 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 -- 172.21.15.76:0/952790396 shutdown_connections 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.380+0000 7f20a0318700 1 -- 172.21.15.76:0/952790396 wait complete. 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f20a0318700 1 Processor -- start 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f20a0318700 1 -- start start 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f20a0318700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f209806ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f20a0318700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2098069f00 con 0x7f20980f6590 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f209e0b4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f209806ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f209e0b4700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f209806ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47782/0 (socket says 172.21.15.76:47782) 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f209e0b4700 1 -- 172.21.15.76:0/1235745399 learned_addr learned my addr 172.21.15.76:0/1235745399 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f209e0b4700 1 -- 172.21.15.76:0/1235745399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2094009d70 con 0x7f20980f6590 2023-12-26T22:04:56.382 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.381+0000 7f209e0b4700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f209806ba80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f2094013f10 tx=0x7f2094009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:56.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.382+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f209400e040 con 0x7f20980f6590 2023-12-26T22:04:56.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.382+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2094007df0 con 0x7f20980f6590 2023-12-26T22:04:56.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.382+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f209806a0a0 con 0x7f20980f6590 2023-12-26T22:04:56.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.382+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2094014d60 con 0x7f20980f6590 2023-12-26T22:04:56.383 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.382+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f209806a530 con 0x7f20980f6590 2023-12-26T22:04:56.385 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.382+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2094007960 con 0x7f20980f6590 2023-12-26T22:04:56.385 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.383+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20980591e0 con 0x7f20980f6590 2023-12-26T22:04:56.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.383+0000 7f208effd700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2084031090 0x7f2084033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:56.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.383+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f20940424e0 con 0x7f20980f6590 2023-12-26T22:04:56.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.383+0000 7f209d8b3700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2084031090 0x7f2084033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:56.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.383+0000 7f209d8b3700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2084031090 0x7f2084033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f20880096a0 tx=0x7f2088006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:56.386 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.388+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2094026c70 con 0x7f20980f6590 2023-12-26T22:04:56.595 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.597+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f20980fc080 con 0x7f20980f6590 2023-12-26T22:04:56.596 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.597+0000 7f208effd700 1 -- 172.21.15.76:0/1235745399 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f209401d320 con 0x7f20980f6590 2023-12-26T22:04:56.596 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:56.596 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2084031090 msgr2=0x7f2084033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2084031090 0x7f2084033550 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f20880096a0 tx=0x7f2088006b40).stop 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 msgr2=0x7f209806ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f209806ba80 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f2094013f10 tx=0x7f2094009ad0).stop 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 shutdown_connections 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2084031090 0x7f2084033550 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 --2- 172.21.15.76:0/1235745399 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f20980f6590 0x7f209806ba80 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 >> 172.21.15.76:0/1235745399 conn(0x7f20980f1cd0 msgr2=0x7f20980f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:56.598 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 shutdown_connections 2023-12-26T22:04:56.599 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:56.599+0000 7f20a0318700 1 -- 172.21.15.76:0/1235745399 wait complete. 2023-12-26T22:04:56.599 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:04:57.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:55.404588+0000 mgr.smithi059.gwkiyc (mgr.14156) 256 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:57.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:56 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:56.598114+0000 mon.smithi059 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.76:0/1235745399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:04:57.980 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:04:57.980 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:04:59.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 -- 172.21.15.76:0/3919585104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 msgr2=0x7f3e800f91f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:59.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 --2- 172.21.15.76:0/3919585104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f91f0 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f3e74009fb0 tx=0x7f3e74009b30).stop 2023-12-26T22:04:59.152 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 -- 172.21.15.76:0/3919585104 shutdown_connections 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 --2- 172.21.15.76:0/3919585104 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f91f0 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 -- 172.21.15.76:0/3919585104 >> 172.21.15.76:0/3919585104 conn(0x7f3e800f0c10 msgr2=0x7f3e800f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 -- 172.21.15.76:0/3919585104 shutdown_connections 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.152+0000 7f3e8572e700 1 -- 172.21.15.76:0/3919585104 wait complete. 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.153+0000 7f3e8572e700 1 Processor -- start 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.153+0000 7f3e8572e700 1 -- start start 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.153+0000 7f3e8572e700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f6c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:59.153 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e8572e700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e800f50f0 con 0x7f3e80060300 2023-12-26T22:04:59.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e7effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f6c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:59.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e7effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f6c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47806/0 (socket says 172.21.15.76:47806) 2023-12-26T22:04:59.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e7effd700 1 -- 172.21.15.76:0/3914260545 learned_addr learned my addr 172.21.15.76:0/3914260545 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:04:59.157 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e7effd700 1 -- 172.21.15.76:0/3914260545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e74005040 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e7effd700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f6c70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f3e74009f80 tx=0x7f3e7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.154+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e74009320 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.155+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3e74013680 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.155+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e800f5290 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.155+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3e7401eda0 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.155+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e800f5720 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.155+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3e74014420 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.155+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e800591e0 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.156+0000 7f3e677fe700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3e68031090 0x7f3e68033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.156+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3e740423f0 con 0x7f3e80060300 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.156+0000 7f3e7e7fc700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3e68031090 0x7f3e68033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:04:59.158 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.156+0000 7f3e7e7fc700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3e68031090 0x7f3e68033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3e700096a0 tx=0x7f3e70006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:04:59.159 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.160+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3e7401e420 con 0x7f3e80060300 2023-12-26T22:04:59.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:58 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:57.404939+0000 mgr.smithi059.gwkiyc (mgr.14156) 257 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:04:59.366 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.367+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3e800fc090 con 0x7f3e80060300 2023-12-26T22:04:59.366 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.368+0000 7f3e677fe700 1 -- 172.21.15.76:0/3914260545 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3e7402c2d0 con 0x7f3e80060300 2023-12-26T22:04:59.367 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:04:59.367 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:04:59.368 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3e68031090 msgr2=0x7f3e68033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:59.368 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3e68031090 0x7f3e68033550 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f3e700096a0 tx=0x7f3e70006b40).stop 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 msgr2=0x7f3e800f6c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f6c70 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f3e74009f80 tx=0x7f3e7400b040).stop 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 shutdown_connections 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3e68031090 0x7f3e68033550 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 --2- 172.21.15.76:0/3914260545 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3e80060300 0x7f3e800f6c70 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 >> 172.21.15.76:0/3914260545 conn(0x7f3e800f0c10 msgr2=0x7f3e800f16f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 shutdown_connections 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:04:59.369+0000 7f3e8572e700 1 -- 172.21.15.76:0/3914260545 wait complete. 2023-12-26T22:04:59.369 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:00.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:04:59 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:04:59.368355+0000 mon.smithi059 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.76:0/3914260545' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:00.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:00.753 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:01.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:00 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:04:59.405199+0000 mgr.smithi059.gwkiyc (mgr.14156) 258 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:02.011 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.012+0000 7f28b1db0700 1 -- 172.21.15.76:0/544515538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 msgr2=0x7f28ac0f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:02.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.012+0000 7f28b1db0700 1 --2- 172.21.15.76:0/544515538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac0f58a0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f289c009fb0 tx=0x7f289c009b30).stop 2023-12-26T22:05:02.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 -- 172.21.15.76:0/544515538 shutdown_connections 2023-12-26T22:05:02.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 --2- 172.21.15.76:0/544515538 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac0f58a0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:02.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 -- 172.21.15.76:0/544515538 >> 172.21.15.76:0/544515538 conn(0x7f28ac0f0be0 msgr2=0x7f28ac0f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:02.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 -- 172.21.15.76:0/544515538 shutdown_connections 2023-12-26T22:05:02.012 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 -- 172.21.15.76:0/544515538 wait complete. 2023-12-26T22:05:02.014 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 Processor -- start 2023-12-26T22:05:02.014 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 -- start start 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac1094f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.013+0000 7f28b1db0700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28ac060320 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac1094f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28ab7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac1094f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47836/0 (socket says 172.21.15.76:47836) 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28ab7fe700 1 -- 172.21.15.76:0/2714152825 learned_addr learned my addr 172.21.15.76:0/2714152825 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28ab7fe700 1 -- 172.21.15.76:0/2714152825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f289c005040 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28ab7fe700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac1094f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f289c0089f0 tx=0x7f289c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f289c009320 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28ac060520 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f289c013ec0 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f289c014d30 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.014+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28ac109ba0 con 0x7f28ac0f5480 2023-12-26T22:05:02.015 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.015+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f289c014420 con 0x7f28ac0f5480 2023-12-26T22:05:02.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.015+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28ac045bb0 con 0x7f28ac0f5480 2023-12-26T22:05:02.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.015+0000 7f2893fff700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2894031090 0x7f2894033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:02.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.015+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f289c0424a0 con 0x7f28ac0f5480 2023-12-26T22:05:02.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.015+0000 7f28aaffd700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2894031090 0x7f2894033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:02.016 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.016+0000 7f28aaffd700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2894031090 0x7f2894033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f28a00096a0 tx=0x7f28a0006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:02.019 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.020+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f289c01d7f0 con 0x7f28ac0f5480 2023-12-26T22:05:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.226+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28ac0faf70 con 0x7f28ac0f5480 2023-12-26T22:05:02.225 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.226+0000 7f2893fff700 1 -- 172.21.15.76:0/2714152825 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f289c01d7f0 con 0x7f28ac0f5480 2023-12-26T22:05:02.226 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:02.226 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2894031090 msgr2=0x7f2894033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2894031090 0x7f2894033550 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f28a00096a0 tx=0x7f28a0006b40).stop 2023-12-26T22:05:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 msgr2=0x7f28ac1094f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac1094f0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f289c0089f0 tx=0x7f289c00b040).stop 2023-12-26T22:05:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 shutdown_connections 2023-12-26T22:05:02.227 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2894031090 0x7f2894033550 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:02.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 --2- 172.21.15.76:0/2714152825 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f28ac0f5480 0x7f28ac1094f0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:02.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 >> 172.21.15.76:0/2714152825 conn(0x7f28ac0f0be0 msgr2=0x7f28ac0f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:02.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 shutdown_connections 2023-12-26T22:05:02.228 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:02.228+0000 7f28b1db0700 1 -- 172.21.15.76:0/2714152825 wait complete. 2023-12-26T22:05:02.228 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:03.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:01.405491+0000 mgr.smithi059.gwkiyc (mgr.14156) 259 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:03.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:02 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:02.227086+0000 mon.smithi059 (mon.0) 304 : audit [DBG] from='client.? 172.21.15.76:0/2714152825' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:03.611 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:03.611 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:04.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.829+0000 7f909dea7700 1 -- 172.21.15.76:0/19674984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 msgr2=0x7f90980f6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:04.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.829+0000 7f909dea7700 1 --2- 172.21.15.76:0/19674984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f90980f6800 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f908c009fb0 tx=0x7f908c009b30).stop 2023-12-26T22:05:04.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.829+0000 7f909dea7700 1 -- 172.21.15.76:0/19674984 shutdown_connections 2023-12-26T22:05:04.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.829+0000 7f909dea7700 1 --2- 172.21.15.76:0/19674984 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f90980f6800 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:04.829 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.829+0000 7f909dea7700 1 -- 172.21.15.76:0/19674984 >> 172.21.15.76:0/19674984 conn(0x7f90980f1b40 msgr2=0x7f90980f3f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:04.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.830+0000 7f909dea7700 1 -- 172.21.15.76:0/19674984 shutdown_connections 2023-12-26T22:05:04.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.830+0000 7f909dea7700 1 -- 172.21.15.76:0/19674984 wait complete. 2023-12-26T22:05:04.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.830+0000 7f909dea7700 1 Processor -- start 2023-12-26T22:05:04.830 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.830+0000 7f909dea7700 1 -- start start 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909dea7700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f9098106620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909dea7700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098106b10 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909cea5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f9098106620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909cea5700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f9098106620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:47842/0 (socket says 172.21.15.76:47842) 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909cea5700 1 -- 172.21.15.76:0/1865941683 learned_addr learned my addr 172.21.15.76:0/1865941683 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909cea5700 1 -- 172.21.15.76:0/1865941683 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908c005040 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.831+0000 7f909cea5700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f9098106620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f908c0089f0 tx=0x7f908c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.832+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908c009320 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.832+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9098106d10 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.832+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f908c013ec0 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.832+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908c014d30 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.832+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90981071a0 con 0x7f90980f63e0 2023-12-26T22:05:04.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.832+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f908c0095f0 con 0x7f90980f63e0 2023-12-26T22:05:04.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.833+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9098045bb0 con 0x7f90980f63e0 2023-12-26T22:05:04.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.833+0000 7f90957fa700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9088031090 0x7f9088033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:04.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.833+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f908c0424a0 con 0x7f90980f63e0 2023-12-26T22:05:04.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.833+0000 7f9097fff700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9088031090 0x7f9088033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:04.833 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.833+0000 7f9097fff700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9088031090 0x7f9088033550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f90840096a0 tx=0x7f9084006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:04.836 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:04.838+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f908c01d980 con 0x7f90980f63e0 2023-12-26T22:05:05.042 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.043+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9098108050 con 0x7f90980f63e0 2023-12-26T22:05:05.043 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.044+0000 7f90957fa700 1 -- 172.21.15.76:0/1865941683 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f908c01d380 con 0x7f90980f63e0 2023-12-26T22:05:05.043 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:05.043 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:05.044 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.045+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9088031090 msgr2=0x7f9088033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.045+0000 7f909dea7700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9088031090 0x7f9088033550 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f90840096a0 tx=0x7f9084006b40).stop 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.045+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 msgr2=0x7f9098106620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.045+0000 7f909dea7700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f9098106620 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f908c0089f0 tx=0x7f908c00b040).stop 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.046+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 shutdown_connections 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.046+0000 7f909dea7700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f9088031090 0x7f9088033550 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.046+0000 7f909dea7700 1 --2- 172.21.15.76:0/1865941683 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f90980f63e0 0x7f9098106620 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.046+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 >> 172.21.15.76:0/1865941683 conn(0x7f90980f1b40 msgr2=0x7f90980f2660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.046+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 shutdown_connections 2023-12-26T22:05:05.045 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:05.046+0000 7f909dea7700 1 -- 172.21.15.76:0/1865941683 wait complete. 2023-12-26T22:05:05.046 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:05.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:04 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:03.405846+0000 mgr.smithi059.gwkiyc (mgr.14156) 260 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:06.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:05 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:05.044682+0000 mon.smithi059 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.76:0/1865941683' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:06.436 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:06.436 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:07.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:06 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:05.406240+0000 mgr.smithi059.gwkiyc (mgr.14156) 261 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:07.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 -- 172.21.15.76:0/1583465859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 msgr2=0x7f3a400f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:07.618 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 --2- 172.21.15.76:0/1583465859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f9160 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f3a34009fb0 tx=0x7f3a34009b30).stop 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 -- 172.21.15.76:0/1583465859 shutdown_connections 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 --2- 172.21.15.76:0/1583465859 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f9160 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 -- 172.21.15.76:0/1583465859 >> 172.21.15.76:0/1583465859 conn(0x7f3a400f0be0 msgr2=0x7f3a400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 -- 172.21.15.76:0/1583465859 shutdown_connections 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.617+0000 7f3a449a9700 1 -- 172.21.15.76:0/1583465859 wait complete. 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a449a9700 1 Processor -- start 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a449a9700 1 -- start start 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a449a9700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a449a9700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a400f5060 con 0x7f3a40060360 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a3e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a3e59c700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:56256/0 (socket says 172.21.15.76:56256) 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a3e59c700 1 -- 172.21.15.76:0/4223610656 learned_addr learned my addr 172.21.15.76:0/4223610656 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:07.619 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a3e59c700 1 -- 172.21.15.76:0/4223610656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a34005040 con 0x7f3a40060360 2023-12-26T22:05:07.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.618+0000 7f3a3e59c700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f6be0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3a34009f80 tx=0x7f3a3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:07.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.619+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a34009320 con 0x7f3a40060360 2023-12-26T22:05:07.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.619+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a400f5200 con 0x7f3a40060360 2023-12-26T22:05:07.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.619+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f3a34013680 con 0x7f3a40060360 2023-12-26T22:05:07.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.619+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3a3401eda0 con 0x7f3a40060360 2023-12-26T22:05:07.620 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.619+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a400f5690 con 0x7f3a40060360 2023-12-26T22:05:07.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.619+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f3a3401e420 con 0x7f3a40060360 2023-12-26T22:05:07.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.620+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a400591e0 con 0x7f3a40060360 2023-12-26T22:05:07.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.620+0000 7f3a2effd700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3a28031090 0x7f3a28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:07.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.620+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f3a34042360 con 0x7f3a40060360 2023-12-26T22:05:07.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.620+0000 7f3a3dd9b700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3a28031090 0x7f3a28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:07.621 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.620+0000 7f3a3dd9b700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3a28031090 0x7f3a28033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3a300096a0 tx=0x7f3a30006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:07.624 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.625+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f3a3401d380 con 0x7f3a40060360 2023-12-26T22:05:07.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.832+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3a400fc060 con 0x7f3a40060360 2023-12-26T22:05:07.832 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.833+0000 7f3a2effd700 1 -- 172.21.15.76:0/4223610656 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f3a3401cb10 con 0x7f3a40060360 2023-12-26T22:05:07.833 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:07.833 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:07.834 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3a28031090 msgr2=0x7f3a28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:07.834 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3a28031090 0x7f3a28033550 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3a300096a0 tx=0x7f3a30006b40).stop 2023-12-26T22:05:07.834 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 msgr2=0x7f3a400f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:07.834 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f6be0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f3a34009f80 tx=0x7f3a3400b040).stop 2023-12-26T22:05:07.834 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 shutdown_connections 2023-12-26T22:05:07.834 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f3a28031090 0x7f3a28033550 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:07.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 --2- 172.21.15.76:0/4223610656 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f3a40060360 0x7f3a400f6be0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:07.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 >> 172.21.15.76:0/4223610656 conn(0x7f3a400f0be0 msgr2=0x7f3a400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:07.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 shutdown_connections 2023-12-26T22:05:07.835 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:07.834+0000 7f3a449a9700 1 -- 172.21.15.76:0/4223610656 wait complete. 2023-12-26T22:05:07.835 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:08.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:07 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:07.834444+0000 mon.smithi059 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.76:0/4223610656' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:09.213 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:09.214 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:09.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:08 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:07.406613+0000 mgr.smithi059.gwkiyc (mgr.14156) 262 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 -- 172.21.15.76:0/4075457634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 msgr2=0x7f690c0f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 --2- 172.21.15.76:0/4075457634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c0f69b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f68fc004760 tx=0x7f68fc009ad0).stop 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 -- 172.21.15.76:0/4075457634 shutdown_connections 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 --2- 172.21.15.76:0/4075457634 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c0f69b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 -- 172.21.15.76:0/4075457634 >> 172.21.15.76:0/4075457634 conn(0x7f690c0f1cd0 msgr2=0x7f690c0f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 -- 172.21.15.76:0/4075457634 shutdown_connections 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.400+0000 7f69140fb700 1 -- 172.21.15.76:0/4075457634 wait complete. 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f69140fb700 1 Processor -- start 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f69140fb700 1 -- start start 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f69140fb700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c06ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:10.402 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f69140fb700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f690c069f00 con 0x7f690c0f6590 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f6911e97700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c06ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f6911e97700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c06ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:56276/0 (socket says 172.21.15.76:56276) 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f6911e97700 1 -- 172.21.15.76:0/1258286320 learned_addr learned my addr 172.21.15.76:0/1258286320 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f6911e97700 1 -- 172.21.15.76:0/1258286320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68fc009d70 con 0x7f690c0f6590 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.401+0000 7f6911e97700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c06ba80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f68fc004170 tx=0x7f68fc009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.402+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68fc00e040 con 0x7f690c0f6590 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.402+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f690c06a0a0 con 0x7f690c0f6590 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.402+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f68fc008ef0 con 0x7f690c0f6590 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.402+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68fc014c90 con 0x7f690c0f6590 2023-12-26T22:05:10.403 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.402+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f690c06a530 con 0x7f690c0f6590 2023-12-26T22:05:10.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.403+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f68fc014420 con 0x7f690c0f6590 2023-12-26T22:05:10.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.403+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f690c0591e0 con 0x7f690c0f6590 2023-12-26T22:05:10.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.403+0000 7f69027fc700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f68f8031090 0x7f68f8033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:10.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.403+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f68fc042240 con 0x7f690c0f6590 2023-12-26T22:05:10.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.403+0000 7f6911696700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f68f8031090 0x7f68f8033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:10.404 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.403+0000 7f6911696700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f68f8031090 0x7f68f8033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f69080096a0 tx=0x7f6908006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:10.407 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.408+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f68fc026300 con 0x7f690c0f6590 2023-12-26T22:05:10.614 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.614+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f690c0fc080 con 0x7f690c0f6590 2023-12-26T22:05:10.615 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.615+0000 7f69027fc700 1 -- 172.21.15.76:0/1258286320 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f68fc012b20 con 0x7f690c0f6590 2023-12-26T22:05:10.615 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:10.615 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f68f8031090 msgr2=0x7f68f8033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f68f8031090 0x7f68f8033550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f69080096a0 tx=0x7f6908006b40).stop 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 msgr2=0x7f690c06ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c06ba80 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f68fc004170 tx=0x7f68fc009ad0).stop 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 shutdown_connections 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f68f8031090 0x7f68f8033550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 --2- 172.21.15.76:0/1258286320 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f690c0f6590 0x7f690c06ba80 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 >> 172.21.15.76:0/1258286320 conn(0x7f690c0f1cd0 msgr2=0x7f690c0f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 shutdown_connections 2023-12-26T22:05:10.617 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:10.617+0000 7f69140fb700 1 -- 172.21.15.76:0/1258286320 wait complete. 2023-12-26T22:05:10.618 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:11.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:09.406987+0000 mgr.smithi059.gwkiyc (mgr.14156) 263 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:11.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:10 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:10.616887+0000 mon.smithi059 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.76:0/1258286320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:12.006 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:12.007 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:13.280 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- 172.21.15.76:0/3849034099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 msgr2=0x7f74c00f9160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 --2- 172.21.15.76:0/3849034099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f9160 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f74b0009fb0 tx=0x7f74b0009b30).stop 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- 172.21.15.76:0/3849034099 shutdown_connections 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 --2- 172.21.15.76:0/3849034099 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f9160 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- 172.21.15.76:0/3849034099 >> 172.21.15.76:0/3849034099 conn(0x7f74c00f0be0 msgr2=0x7f74c00f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- 172.21.15.76:0/3849034099 shutdown_connections 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- 172.21.15.76:0/3849034099 wait complete. 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 Processor -- start 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- start start 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f6be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.280+0000 7f74c8099700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74c00f5060 con 0x7f74c0060360 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c5e35700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f6be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:13.281 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c5e35700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f6be0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:56292/0 (socket says 172.21.15.76:56292) 2023-12-26T22:05:13.282 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c5e35700 1 -- 172.21.15.76:0/3517490790 learned_addr learned my addr 172.21.15.76:0/3517490790 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:13.282 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c5e35700 1 -- 172.21.15.76:0/3517490790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b0005040 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c5e35700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f6be0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f74b0009f80 tx=0x7f74b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b0009320 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f74b0013680 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74c00f5200 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74b001eda0 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.281+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74c00f5690 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.282+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f74b0014420 con 0x7f74c0060360 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.282+0000 7f74b67fc700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f74ac031090 0x7f74ac033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:13.283 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.282+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f74b0042420 con 0x7f74c0060360 2023-12-26T22:05:13.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.282+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74c00591e0 con 0x7f74c0060360 2023-12-26T22:05:13.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.283+0000 7f74c5634700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f74ac031090 0x7f74ac033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:13.284 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.283+0000 7f74c5634700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f74ac031090 0x7f74ac033550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f74bc0096a0 tx=0x7f74bc006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:13.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.287+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f74b0014700 con 0x7f74c0060360 2023-12-26T22:05:13.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:12 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:11.407366+0000 mgr.smithi059.gwkiyc (mgr.14156) 264 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:13.496 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.496+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f74c0045bb0 con 0x7f74c0060360 2023-12-26T22:05:13.497 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.497+0000 7f74b67fc700 1 -- 172.21.15.76:0/3517490790 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f74b0026350 con 0x7f74c0060360 2023-12-26T22:05:13.497 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:13.497 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f74ac031090 msgr2=0x7f74ac033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f74ac031090 0x7f74ac033550 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f74bc0096a0 tx=0x7f74bc006b40).stop 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 msgr2=0x7f74c00f6be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f6be0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f74b0009f80 tx=0x7f74b000b040).stop 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 shutdown_connections 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f74ac031090 0x7f74ac033550 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 --2- 172.21.15.76:0/3517490790 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f74c0060360 0x7f74c00f6be0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 >> 172.21.15.76:0/3517490790 conn(0x7f74c00f0be0 msgr2=0x7f74c00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 shutdown_connections 2023-12-26T22:05:13.499 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:13.499+0000 7f74c8099700 1 -- 172.21.15.76:0/3517490790 wait complete. 2023-12-26T22:05:13.500 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:14.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:13 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:13.498838+0000 mon.smithi059 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.76:0/3517490790' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:14.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:14.885 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:15.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:14 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:13.407712+0000 mgr.smithi059.gwkiyc (mgr.14156) 265 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:16.074 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 -- 172.21.15.76:0/554358231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 msgr2=0x7f2f400f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 --2- 172.21.15.76:0/554358231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400f58a0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f2f34009fb0 tx=0x7f2f34009b30).stop 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 -- 172.21.15.76:0/554358231 shutdown_connections 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 --2- 172.21.15.76:0/554358231 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400f58a0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 -- 172.21.15.76:0/554358231 >> 172.21.15.76:0/554358231 conn(0x7f2f400f0be0 msgr2=0x7f2f400f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 -- 172.21.15.76:0/554358231 shutdown_connections 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 -- 172.21.15.76:0/554358231 wait complete. 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.074+0000 7f2f45efe700 1 Processor -- start 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f45efe700 1 -- start start 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f45efe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f45efe700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f400fb8e0 con 0x7f2f400f5480 2023-12-26T22:05:16.075 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f3f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:16.076 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f3f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52196/0 (socket says 172.21.15.76:52196) 2023-12-26T22:05:16.076 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f3f7fe700 1 -- 172.21.15.76:0/4226670974 learned_addr learned my addr 172.21.15.76:0/4226670974 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:16.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f3f7fe700 1 -- 172.21.15.76:0/4226670974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f34005040 con 0x7f2f400f5480 2023-12-26T22:05:16.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f3f7fe700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400fd460 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f2f340089f0 tx=0x7f2f3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:16.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.075+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f34009320 con 0x7f2f400f5480 2023-12-26T22:05:16.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f400fba80 con 0x7f2f400f5480 2023-12-26T22:05:16.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f2f34013ec0 con 0x7f2f400f5480 2023-12-26T22:05:16.077 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f34014d30 con 0x7f2f400f5480 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f400fbf10 con 0x7f2f400f5480 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f2f34014420 con 0x7f2f400f5480 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f400591e0 con 0x7f2f400f5480 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.076+0000 7f2f27fff700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f28031090 0x7f2f28033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.077+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f2f34042400 con 0x7f2f400f5480 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.077+0000 7f2f3effd700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f28031090 0x7f2f28033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:16.078 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.077+0000 7f2f3effd700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f28031090 0x7f2f28033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f2f300096a0 tx=0x7f2f30006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:16.081 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.081+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f2f340263a0 con 0x7f2f400f5480 2023-12-26T22:05:16.290 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.290+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f400faf70 con 0x7f2f400f5480 2023-12-26T22:05:16.290 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.291+0000 7f2f27fff700 1 -- 172.21.15.76:0/4226670974 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f2f3401d380 con 0x7f2f400f5480 2023-12-26T22:05:16.291 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:16.291 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:16.292 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f28031090 msgr2=0x7f2f28033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:16.292 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f28031090 0x7f2f28033550 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f2f300096a0 tx=0x7f2f30006b40).stop 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 msgr2=0x7f2f400fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400fd460 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f2f340089f0 tx=0x7f2f3400b040).stop 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 shutdown_connections 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f2f28031090 0x7f2f28033550 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 --2- 172.21.15.76:0/4226670974 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f2f400f5480 0x7f2f400fd460 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.292+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 >> 172.21.15.76:0/4226670974 conn(0x7f2f400f0be0 msgr2=0x7f2f4005fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.293+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 shutdown_connections 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:16.293+0000 7f2f45efe700 1 -- 172.21.15.76:0/4226670974 wait complete. 2023-12-26T22:05:16.293 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:17.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:15.408004+0000 mgr.smithi059.gwkiyc (mgr.14156) 266 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:17.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:16 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:16.292571+0000 mon.smithi059 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.76:0/4226670974' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:17.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:17.674 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 -- 172.21.15.76:0/2232079632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 msgr2=0x7f58400f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 --2- 172.21.15.76:0/2232079632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f58400f69b0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f5838009a00 tx=0x7f5838004cb0).stop 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 -- 172.21.15.76:0/2232079632 shutdown_connections 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 --2- 172.21.15.76:0/2232079632 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f58400f69b0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 -- 172.21.15.76:0/2232079632 >> 172.21.15.76:0/2232079632 conn(0x7f58400f1cd0 msgr2=0x7f58400f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 -- 172.21.15.76:0/2232079632 shutdown_connections 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.904+0000 7f5845f7f700 1 -- 172.21.15.76:0/2232079632 wait complete. 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f5845f7f700 1 Processor -- start 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f5845f7f700 1 -- start start 2023-12-26T22:05:18.905 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f5845f7f700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f584006ba80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:18.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f5845f7f700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5840069f00 con 0x7f58400f6590 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f584006ba80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f583f7fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f584006ba80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52204/0 (socket says 172.21.15.76:52204) 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.905+0000 7f583f7fe700 1 -- 172.21.15.76:0/4073645339 learned_addr learned my addr 172.21.15.76:0/4073645339 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f583f7fe700 1 -- 172.21.15.76:0/4073645339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5838009640 con 0x7f58400f6590 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f583f7fe700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f584006ba80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f5838009610 tx=0x7f5838004cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f58380092e0 con 0x7f58400f6590 2023-12-26T22:05:18.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f5838007d50 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f584006a0a0 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5838003de0 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.906+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f584006a530 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.907+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58400591e0 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.907+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f5838004030 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.907+0000 7f5837fff700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f582c031090 0x7f582c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.907+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f5838042d40 con 0x7f58400f6590 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.907+0000 7f583effd700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f582c031090 0x7f582c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:18.908 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.908+0000 7f583effd700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f582c031090 0x7f582c033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f58280096a0 tx=0x7f5828006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:18.912 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:18.912+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f58380262f0 con 0x7f58400f6590 2023-12-26T22:05:19.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.117+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f58400fc080 con 0x7f58400f6590 2023-12-26T22:05:19.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.118+0000 7f5837fff700 1 -- 172.21.15.76:0/4073645339 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f58380262f0 con 0x7f58400f6590 2023-12-26T22:05:19.118 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:19.118 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:19.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.119+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f582c031090 msgr2=0x7f582c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.119+0000 7f5845f7f700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f582c031090 0x7f582c033550 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f58280096a0 tx=0x7f5828006b40).stop 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.119+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 msgr2=0x7f584006ba80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.119+0000 7f5845f7f700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f584006ba80 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f5838009610 tx=0x7f5838004cb0).stop 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.120+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 shutdown_connections 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.120+0000 7f5845f7f700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f582c031090 0x7f582c033550 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.120+0000 7f5845f7f700 1 --2- 172.21.15.76:0/4073645339 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f58400f6590 0x7f584006ba80 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.120+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 >> 172.21.15.76:0/4073645339 conn(0x7f58400f1cd0 msgr2=0x7f58400f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.120+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 shutdown_connections 2023-12-26T22:05:19.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:19.120+0000 7f5845f7f700 1 -- 172.21.15.76:0/4073645339 wait complete. 2023-12-26T22:05:19.121 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:19.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:18 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:17.408384+0000 mgr.smithi059.gwkiyc (mgr.14156) 267 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:20.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:19 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:19.119439+0000 mon.smithi059 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.76:0/4073645339' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:20.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:20.501 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:21.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:20 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:19.408681+0000 mgr.smithi059.gwkiyc (mgr.14156) 268 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 -- 172.21.15.76:0/2887512049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 msgr2=0x7fd8e40f69b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 --2- 172.21.15.76:0/2887512049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e40f69b0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fd8d4004760 tx=0x7fd8d4009ad0).stop 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 -- 172.21.15.76:0/2887512049 shutdown_connections 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 --2- 172.21.15.76:0/2887512049 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e40f69b0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 -- 172.21.15.76:0/2887512049 >> 172.21.15.76:0/2887512049 conn(0x7fd8e40f1cd0 msgr2=0x7fd8e40f4110 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 -- 172.21.15.76:0/2887512049 shutdown_connections 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.686+0000 7fd8e9182700 1 -- 172.21.15.76:0/2887512049 wait complete. 2023-12-26T22:05:21.688 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e9182700 1 Processor -- start 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e9182700 1 -- start start 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e9182700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e41067c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e9182700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8e4106cb0 con 0x7fd8e40f6590 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e41067c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e2d9d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e41067c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52222/0 (socket says 172.21.15.76:52222) 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e2d9d700 1 -- 172.21.15.76:0/4180882991 learned_addr learned my addr 172.21.15.76:0/4180882991 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e2d9d700 1 -- 172.21.15.76:0/4180882991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8d4009d70 con 0x7fd8e40f6590 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.687+0000 7fd8e2d9d700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e41067c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fd8d4004170 tx=0x7fd8d4009ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.688+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d400e040 con 0x7fd8e40f6590 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.688+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8e4106eb0 con 0x7fd8e40f6590 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.688+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd8d4008ef0 con 0x7fd8e40f6590 2023-12-26T22:05:21.689 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.688+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8d4014c90 con 0x7fd8e40f6590 2023-12-26T22:05:21.690 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.688+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8e4107340 con 0x7fd8e40f6590 2023-12-26T22:05:21.690 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.689+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd8d4014420 con 0x7fd8e40f6590 2023-12-26T22:05:21.690 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.689+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8e4045bb0 con 0x7fd8e40f6590 2023-12-26T22:05:21.691 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.689+0000 7fd8cb7fe700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd8cc031090 0x7fd8cc033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:21.691 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.689+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd8d40421f0 con 0x7fd8e40f6590 2023-12-26T22:05:21.691 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.689+0000 7fd8e259c700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd8cc031090 0x7fd8cc033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:21.691 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.690+0000 7fd8e259c700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd8cc031090 0x7fd8cc033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd8d80096a0 tx=0x7fd8d8006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:21.693 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.694+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd8d4026300 con 0x7fd8e40f6590 2023-12-26T22:05:21.904 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.904+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8e41081f0 con 0x7fd8e40f6590 2023-12-26T22:05:21.904 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.905+0000 7fd8cb7fe700 1 -- 172.21.15.76:0/4180882991 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd8d4012b20 con 0x7fd8e40f6590 2023-12-26T22:05:21.905 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:21.905 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd8cc031090 msgr2=0x7fd8cc033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd8cc031090 0x7fd8cc033550 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fd8d80096a0 tx=0x7fd8d8006b40).stop 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 msgr2=0x7fd8e41067c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e41067c0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fd8d4004170 tx=0x7fd8d4009ad0).stop 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 shutdown_connections 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd8cc031090 0x7fd8cc033550 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 --2- 172.21.15.76:0/4180882991 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd8e40f6590 0x7fd8e41067c0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 >> 172.21.15.76:0/4180882991 conn(0x7fd8e40f1cd0 msgr2=0x7fd8e40f27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:21.906 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 shutdown_connections 2023-12-26T22:05:21.907 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:21.906+0000 7fd8e9182700 1 -- 172.21.15.76:0/4180882991 wait complete. 2023-12-26T22:05:21.907 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:23.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:21.409042+0000 mgr.smithi059.gwkiyc (mgr.14156) 269 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:23.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:22 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:21.906313+0000 mon.smithi059 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.76:0/4180882991' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:23.297 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:23.297 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:24.507 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 -- 172.21.15.76:0/3872905933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 msgr2=0x7f84880f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 --2- 172.21.15.76:0/3872905933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84880f58a0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f8480009960 tx=0x7f8480004d10).stop 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 -- 172.21.15.76:0/3872905933 shutdown_connections 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 --2- 172.21.15.76:0/3872905933 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84880f58a0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 -- 172.21.15.76:0/3872905933 >> 172.21.15.76:0/3872905933 conn(0x7f84880f0be0 msgr2=0x7f84880f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 -- 172.21.15.76:0/3872905933 shutdown_connections 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.507+0000 7f848db97700 1 -- 172.21.15.76:0/3872905933 wait complete. 2023-12-26T22:05:24.508 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f848db97700 1 Processor -- start 2023-12-26T22:05:24.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f848db97700 1 -- start start 2023-12-26T22:05:24.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f848db97700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84881056f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:24.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f848db97700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488105be0 con 0x7f84880f5480 2023-12-26T22:05:24.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f84877fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84881056f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:24.510 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f84877fe700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84881056f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:52230/0 (socket says 172.21.15.76:52230) 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.508+0000 7f84877fe700 1 -- 172.21.15.76:0/2246876745 learned_addr learned my addr 172.21.15.76:0/2246876745 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f84877fe700 1 -- 172.21.15.76:0/2246876745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8480009640 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f84877fe700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84881056f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f8480009610 tx=0x7f8480004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f848001a070 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8488105de0 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f848000b7e0 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f848001f600 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.509+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8488106270 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.510+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f8480007620 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.510+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8488045bb0 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.510+0000 7f847ffff700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8474031090 0x7f8474033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.510+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f8480044d90 con 0x7f84880f5480 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.510+0000 7f8486ffd700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8474031090 0x7f8474033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:24.511 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.511+0000 7f8486ffd700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8474031090 0x7f8474033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f84700096a0 tx=0x7f8470006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:24.515 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.515+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f848001ddb0 con 0x7f84880f5480 2023-12-26T22:05:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.721+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84880faf70 con 0x7f84880f5480 2023-12-26T22:05:24.721 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.722+0000 7f847ffff700 1 -- 172.21.15.76:0/2246876745 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f8480011320 con 0x7f84880f5480 2023-12-26T22:05:24.722 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:24.722 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8474031090 msgr2=0x7f8474033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8474031090 0x7f8474033550 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f84700096a0 tx=0x7f8470006b40).stop 2023-12-26T22:05:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 msgr2=0x7f84881056f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84881056f0 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f8480009610 tx=0x7f8480004d10).stop 2023-12-26T22:05:24.723 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 shutdown_connections 2023-12-26T22:05:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f8474031090 0x7f8474033550 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 --2- 172.21.15.76:0/2246876745 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f84880f5480 0x7f84881056f0 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 >> 172.21.15.76:0/2246876745 conn(0x7f84880f0be0 msgr2=0x7f84880f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 shutdown_connections 2023-12-26T22:05:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:24.723+0000 7f848db97700 1 -- 172.21.15.76:0/2246876745 wait complete. 2023-12-26T22:05:24.724 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:25.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:23.409378+0000 mgr.smithi059.gwkiyc (mgr.14156) 270 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:25.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:24 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:24.723417+0000 mon.smithi059 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.76:0/2246876745' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:26.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:26.107 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:27.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 -- 172.21.15.76:0/3988053994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 msgr2=0x7f85400f58d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:27.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 --2- 172.21.15.76:0/3988053994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85400f58d0 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7f8528009fb0 tx=0x7f8528009b30).stop 2023-12-26T22:05:27.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 -- 172.21.15.76:0/3988053994 shutdown_connections 2023-12-26T22:05:27.286 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 --2- 172.21.15.76:0/3988053994 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85400f58d0 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 -- 172.21.15.76:0/3988053994 >> 172.21.15.76:0/3988053994 conn(0x7f85400f0c10 msgr2=0x7f85400f3030 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 -- 172.21.15.76:0/3988053994 shutdown_connections 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.285+0000 7f8545752700 1 -- 172.21.15.76:0/3988053994 wait complete. 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f8545752700 1 Processor -- start 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f8545752700 1 -- start start 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f8545752700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85401067f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f8545752700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8540106ce0 con 0x7f85400f54b0 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f853effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85401067f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f853effd700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85401067f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:37674/0 (socket says 172.21.15.76:37674) 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.286+0000 7f853effd700 1 -- 172.21.15.76:0/91137336 learned_addr learned my addr 172.21.15.76:0/91137336 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f853effd700 1 -- 172.21.15.76:0/91137336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8528005040 con 0x7f85400f54b0 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f853effd700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85401067f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f85280089f0 tx=0x7f852800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:27.287 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8528009320 con 0x7f85400f54b0 2023-12-26T22:05:27.288 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8540106ee0 con 0x7f85400f54b0 2023-12-26T22:05:27.288 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7f8528013ec0 con 0x7f85400f54b0 2023-12-26T22:05:27.288 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8528014d30 con 0x7f85400f54b0 2023-12-26T22:05:27.288 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.287+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8540107370 con 0x7f85400f54b0 2023-12-26T22:05:27.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.288+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7f85280263a0 con 0x7f85400f54b0 2023-12-26T22:05:27.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.288+0000 7f85377fe700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f852c031090 0x7f852c033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:27.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.288+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7f85280423f0 con 0x7f85400f54b0 2023-12-26T22:05:27.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.288+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8540045bb0 con 0x7f85400f54b0 2023-12-26T22:05:27.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.288+0000 7f853e7fc700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f852c031090 0x7f852c033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:27.289 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.289+0000 7f853e7fc700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f852c031090 0x7f852c033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f85300096a0 tx=0x7f8530006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:27.292 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.293+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7f85280266d0 con 0x7f85400f54b0 2023-12-26T22:05:27.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:26 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:25.409742+0000 mgr.smithi059.gwkiyc (mgr.14156) 271 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:27.501 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.502+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8540107ff0 con 0x7f85400f54b0 2023-12-26T22:05:27.502 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.503+0000 7f85377fe700 1 -- 172.21.15.76:0/91137336 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7f852802b020 con 0x7f85400f54b0 2023-12-26T22:05:27.502 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:27.502 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f852c031090 msgr2=0x7f852c033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f852c031090 0x7f852c033550 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f85300096a0 tx=0x7f8530006b40).stop 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 msgr2=0x7f85401067f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85401067f0 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f85280089f0 tx=0x7f852800b040).stop 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 shutdown_connections 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7f852c031090 0x7f852c033550 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 --2- 172.21.15.76:0/91137336 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7f85400f54b0 0x7f85401067f0 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 >> 172.21.15.76:0/91137336 conn(0x7f85400f0c10 msgr2=0x7f85400f16d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 shutdown_connections 2023-12-26T22:05:27.504 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:27.505+0000 7f8545752700 1 -- 172.21.15.76:0/91137336 wait complete. 2023-12-26T22:05:27.505 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:28.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:27 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:27.503678+0000 mon.smithi059 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.76:0/91137336' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:28.883 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-12-26T22:05:28.883 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/daemon-base:latest-pacific shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 -- ceph mon dump -f json 2023-12-26T22:05:29.293 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:28 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:27.410070+0000 mgr.smithi059.gwkiyc (mgr.14156) 272 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:30.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 -- 172.21.15.76:0/876213883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 msgr2=0x7fd7140f58a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:30.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 --2- 172.21.15.76:0/876213883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140f58a0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fd704009fb0 tx=0x7fd704009b30).stop 2023-12-26T22:05:30.116 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 -- 172.21.15.76:0/876213883 shutdown_connections 2023-12-26T22:05:30.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 --2- 172.21.15.76:0/876213883 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140f58a0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:30.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 -- 172.21.15.76:0/876213883 >> 172.21.15.76:0/876213883 conn(0x7fd7140f0be0 msgr2=0x7fd7140f3000 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:30.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 -- 172.21.15.76:0/876213883 shutdown_connections 2023-12-26T22:05:30.117 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.117+0000 7fd71c5e1700 1 -- 172.21.15.76:0/876213883 wait complete. 2023-12-26T22:05:30.118 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71c5e1700 1 Processor -- start 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71c5e1700 1 -- start start 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71c5e1700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140fd460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71c5e1700 1 -- --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7140fb8e0 con 0x7fd7140f5480 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71a37d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140fd460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71a37d700 1 --2- >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140fd460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.59:3300/0 says I am v2:172.21.15.76:37688/0 (socket says 172.21.15.76:37688) 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.118+0000 7fd71a37d700 1 -- 172.21.15.76:0/234419706 learned_addr learned my addr 172.21.15.76:0/234419706 (peer_addr_for_me v2:172.21.15.76:0/0) 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd71a37d700 1 -- 172.21.15.76:0/234419706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd704005040 con 0x7fd7140f5480 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd71a37d700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140fd460 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fd7040089f0 tx=0x7fd70400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd704009320 con 0x7fd7140f5480 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7140fba80 con 0x7fd7140f5480 2023-12-26T22:05:30.119 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 2 ==== config(25 keys) v1 ==== 1054+0+0 (secure 0 0 0) 0x7fd704013ec0 con 0x7fd7140f5480 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd704014d30 con 0x7fd7140f5480 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.119+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7140fbf10 con 0x7fd7140f5480 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.120+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 4 ==== mgrmap(e 13) v1 ==== 37151+0+0 (secure 0 0 0) 0x7fd7040095f0 con 0x7fd7140f5480 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.120+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7140591e0 con 0x7fd7140f5480 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.120+0000 7fd70affd700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd700031090 0x7fd700033550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.120+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1065+0+0 (secure 0 0 0) 0x7fd7040423e0 con 0x7fd7140f5480 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.120+0000 7fd719b7c700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd700031090 0x7fd700033550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-12-26T22:05:30.120 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.121+0000 7fd719b7c700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd700031090 0x7fd700033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fd7100096a0 tx=0x7fd710006b40).ready entity=mgr.14156 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-12-26T22:05:30.123 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.125+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+153237 (secure 0 0 0) 0x7fd7040268c0 con 0x7fd7140f5480 2023-12-26T22:05:30.328 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.329+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 --> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd7140faf70 con 0x7fd7140f5480 2023-12-26T22:05:30.329 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.330+0000 7fd70affd700 1 -- 172.21.15.76:0/234419706 <== mon.0 v2:172.21.15.59:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+690 (secure 0 0 0) 0x7fd70401d7f0 con 0x7fd7140f5480 2023-12-26T22:05:30.329 INFO:teuthology.orchestra.run.smithi076.stdout: 2023-12-26T22:05:30.330 INFO:teuthology.orchestra.run.smithi076.stdout:{"epoch":1,"fsid":"761f1ff0-a439-11ee-95a5-87774f69a715","modified":"2023-12-26T21:55:56.269247Z","created":"2023-12-26T21:55:56.269247Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi059","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.59:3300","nonce":0},{"type":"v1","addr":"172.21.15.59:6789","nonce":0}]},"addr":"172.21.15.59:6789/0","public_addr":"172.21.15.59:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-12-26T22:05:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd700031090 msgr2=0x7fd700033550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd700031090 0x7fd700033550 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fd7100096a0 tx=0x7fd710006b40).stop 2023-12-26T22:05:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 msgr2=0x7fd7140fd460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-12-26T22:05:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140fd460 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fd7040089f0 tx=0x7fd70400b040).stop 2023-12-26T22:05:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 shutdown_connections 2023-12-26T22:05:30.331 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:6800/1607822224,v1:172.21.15.59:6801/1607822224] conn(0x7fd700031090 0x7fd700033550 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 --2- 172.21.15.76:0/234419706 >> [v2:172.21.15.59:3300/0,v1:172.21.15.59:6789/0] conn(0x7fd7140f5480 0x7fd7140fd460 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-12-26T22:05:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 >> 172.21.15.76:0/234419706 conn(0x7fd7140f0be0 msgr2=0x7fd71405fee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-12-26T22:05:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 shutdown_connections 2023-12-26T22:05:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:2023-12-26T22:05:30.332+0000 7fd71c5e1700 1 -- 172.21.15.76:0/234419706 wait complete. 2023-12-26T22:05:30.332 INFO:teuthology.orchestra.run.smithi076.stderr:dumped monmap epoch 1 2023-12-26T22:05:30.717 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-12-26T22:05:30.718 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-12-26T22:05:30.719 DEBUG:teuthology.orchestra.run.smithi059:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-26T22:05:30.742 DEBUG:teuthology.orchestra.run.smithi076:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-12-26T22:05:30.760 INFO:tasks.cephadm:Stopping all daemons... 2023-12-26T22:05:30.761 INFO:tasks.cephadm.mon.smithi059:Stopping mon.smithi059... 2023-12-26T22:05:30.761 DEBUG:teuthology.orchestra.run.smithi059:> sudo systemctl stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059 2023-12-26T22:05:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:30 smithi059 systemd[1]: Stopping Ceph mon.smithi059 for 761f1ff0-a439-11ee-95a5-87774f69a715... 2023-12-26T22:05:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:30 smithi059 podman[117933]: Error: no container with name or ID "ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon.smithi059" found: no such container 2023-12-26T22:05:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: cluster 2023-12-26T22:05:29.410359+0000 mgr.smithi059.gwkiyc (mgr.14156) 273 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-12-26T22:05:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:30 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: audit 2023-12-26T22:05:30.330913+0000 mon.smithi059 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.76:0/234419706' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-12-26T22:05:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T22:05:31.031+0000 7ff629287700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi059 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-12-26T22:05:31.294 INFO:journalctl@ceph.mon.smithi059.smithi059.stdout:Dec 26 22:05:31 smithi059 ceph-761f1ff0-a439-11ee-95a5-87774f69a715-mon-smithi059[111931]: debug 2023-12-26T22:05:31.031+0000 7ff629287700 -1 mon.smithi059@0(leader) e1 *** Got Signal Terminated *** 2023-12-26T22:05:31.519 DEBUG:teuthology.orchestra.run.smithi059:> sudo pkill -f 'journalctl -f -n 0 -u ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi059.service' 2023-12-26T22:05:31.567 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-12-26T22:05:31.568 INFO:tasks.cephadm.mon.smithi059:Stopped mon.smithi059 2023-12-26T22:05:31.568 INFO:tasks.cephadm.mon.smithi076:Stopping mon.smithi076... 2023-12-26T22:05:31.568 DEBUG:teuthology.orchestra.run.smithi076:> sudo systemctl stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076 2023-12-26T22:05:31.605 INFO:teuthology.orchestra.run.smithi076.stderr:Failed to stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076.service: Unit ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076.service not loaded. 2023-12-26T22:05:31.606 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-12-26T22:05:31.607 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi076" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 5: 'sudo systemctl stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076' 2023-12-26T22:05:31.608 INFO:tasks.cephadm:Archiving crash dumps... 2023-12-26T22:05:31.610 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/remote/smithi059/crash 2023-12-26T22:05:31.611 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/crash -- . 2023-12-26T22:05:31.651 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/crash to /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/remote/smithi076/crash 2023-12-26T22:05:31.652 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/crash -- . 2023-12-26T22:05:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:tar: /var/lib/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/crash: Cannot open: No such file or directory 2023-12-26T22:05:31.698 INFO:teuthology.orchestra.run.smithi076.stderr:tar: Error is not recoverable: exiting now 2023-12-26T22:05:31.699 INFO:tasks.cephadm:Checking cluster log for badness... 2023-12-26T22:05:31.700 DEBUG:teuthology.orchestra.run.smithi059:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v '\(POOL_APP_NOT_ENABLED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-12-26T22:05:31.729 INFO:teuthology.orchestra.run.smithi059.stderr:grep: /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph.log: No such file or directory 2023-12-26T22:05:31.731 INFO:tasks.cephadm:Compressing logs... 2023-12-26T22:05:31.731 DEBUG:teuthology.orchestra.run.smithi059:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-26T22:05:31.774 DEBUG:teuthology.orchestra.run.smithi076:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-12-26T22:05:31.798 INFO:teuthology.orchestra.run.smithi059.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2023-12-26T22:05:31.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-26T22:05:31.799 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-mon.smithi059.log 2023-12-26T22:05:31.800 INFO:teuthology.orchestra.run.smithi059.stderr:gzip -5 --verbose -- /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-volume.log 2023-12-26T22:05:31.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-mgr.smithi059.gwkiyc.log 2023-12-26T22:05:31.800 INFO:teuthology.orchestra.run.smithi059.stderr:/var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-mon.smithi059.log: /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-volume.log: /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-mgr.smithi059.gwkiyc.log: 90.2% -- replaced with /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-volume.log.gz 2023-12-26T22:05:31.804 INFO:teuthology.orchestra.run.smithi076.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-12-26T22:05:31.804 INFO:teuthology.orchestra.run.smithi076.stderr:: No such file or directory 2023-12-26T22:05:31.805 INFO:teuthology.orchestra.run.smithi059.stderr: 92.7% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-26T22:05:31.809 INFO:teuthology.orchestra.run.smithi076.stderr:/var/log/ceph/cephadm.log: 95.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-12-26T22:05:31.811 INFO:teuthology.orchestra.run.smithi076.stderr: 2023-12-26T22:05:31.811 INFO:teuthology.orchestra.run.smithi076.stderr:real 0m0.017s 2023-12-26T22:05:31.811 INFO:teuthology.orchestra.run.smithi076.stderr:user 0m0.016s 2023-12-26T22:05:31.811 INFO:teuthology.orchestra.run.smithi076.stderr:sys 0m0.011s 2023-12-26T22:05:31.873 INFO:teuthology.orchestra.run.smithi059.stderr: 92.8% -- replaced with /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-mgr.smithi059.gwkiyc.log.gz 2023-12-26T22:05:31.946 INFO:teuthology.orchestra.run.smithi059.stderr: 90.4% -- replaced with /var/log/ceph/761f1ff0-a439-11ee-95a5-87774f69a715/ceph-mon.smithi059.log.gz 2023-12-26T22:05:31.948 INFO:teuthology.orchestra.run.smithi059.stderr: 2023-12-26T22:05:31.948 INFO:teuthology.orchestra.run.smithi059.stderr:real 0m0.159s 2023-12-26T22:05:31.948 INFO:teuthology.orchestra.run.smithi059.stderr:user 0m0.231s 2023-12-26T22:05:31.948 INFO:teuthology.orchestra.run.smithi059.stderr:sys 0m0.017s 2023-12-26T22:05:31.948 INFO:tasks.cephadm:Archiving logs... 2023-12-26T22:05:31.948 DEBUG:teuthology.misc:Transferring archived files from smithi059:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/remote/smithi059/log 2023-12-26T22:05:31.949 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-26T22:05:32.058 DEBUG:teuthology.misc:Transferring archived files from smithi076:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/remote/smithi076/log 2023-12-26T22:05:32.058 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /var/log/ceph -- . 2023-12-26T22:05:32.091 INFO:tasks.cephadm:Removing cluster... 2023-12-26T22:05:32.091 DEBUG:teuthology.orchestra.run.smithi059:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 --force 2023-12-26T22:06:43.038 DEBUG:teuthology.orchestra.run.smithi076:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 761f1ff0-a439-11ee-95a5-87774f69a715 --force 2023-12-26T22:06:43.494 INFO:tasks.cephadm:Removing cephadm ... 2023-12-26T22:06:43.494 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-26T22:06:43.516 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf /home/ubuntu/cephtest/cephadm 2023-12-26T22:06:43.535 INFO:tasks.cephadm:Teardown complete 2023-12-26T22:06:43.535 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 5: 'sudo systemctl stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076' 2023-12-26T22:06:43.615 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=a673131c2a824828b9d03caf51701a00 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 549, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 675, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 1564, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 1477, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_5b5a60d2d510bfaa9213c41e32227fb008245a30/qa/tasks/cephadm.py", line 568, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_9680d2aba1e76f64ade295c81599c97907457848/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi076 with status 5: 'sudo systemctl stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076' 2023-12-26T22:06:43.619 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-12-26T22:06:43.629 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-12-26T22:06:43.676 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-12-26T22:06:43.677 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-26T22:06:43.690 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-12-26T22:06:43.771 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-12-26T22:06:43.780 INFO:teuthology.task.clock:Checking final clock skew... 2023-12-26T22:06:43.780 DEBUG:teuthology.orchestra.run.smithi059:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-26T22:06:43.783 DEBUG:teuthology.orchestra.run.smithi076:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-12-26T22:06:43.800 INFO:teuthology.orchestra.run.smithi076.stderr:bash: ntpq: command not found 2023-12-26T22:06:43.801 INFO:teuthology.orchestra.run.smithi059.stderr:bash: ntpq: command not found 2023-12-26T22:06:43.804 INFO:teuthology.orchestra.run.smithi076.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-26T22:06:43.804 INFO:teuthology.orchestra.run.smithi076.stdout:=============================================================================== 2023-12-26T22:06:43.804 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 10 +657us[ +657us] +/- 93ms 2023-12-26T22:06:43.804 INFO:teuthology.orchestra.run.smithi076.stdout:^* hv02.front.sepia.ceph.com 3 6 377 11 +419us[ +421us] +/- 53ms 2023-12-26T22:06:43.804 INFO:teuthology.orchestra.run.smithi076.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 9 -1080us[-1080us] +/- 73ms 2023-12-26T22:06:43.805 INFO:teuthology.orchestra.run.smithi076.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T22:06:43.806 INFO:teuthology.orchestra.run.smithi059.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-12-26T22:06:43.806 INFO:teuthology.orchestra.run.smithi059.stdout:=============================================================================== 2023-12-26T22:06:43.806 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 9 +707us[ +713us] +/- 93ms 2023-12-26T22:06:43.807 INFO:teuthology.orchestra.run.smithi059.stdout:^* hv02.front.sepia.ceph.com 3 6 377 8 +419us[ +425us] +/- 53ms 2023-12-26T22:06:43.807 INFO:teuthology.orchestra.run.smithi059.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 10 -1037us[-1031us] +/- 73ms 2023-12-26T22:06:43.807 INFO:teuthology.orchestra.run.smithi059.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-12-26T22:06:43.808 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-12-26T22:06:43.817 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-12-26T22:06:43.817 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-12-26T22:06:43.845 DEBUG:teuthology.orchestra.run.smithi059:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-26T22:06:43.945 DEBUG:teuthology.orchestra.run.smithi076:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-12-26T22:06:44.035 DEBUG:teuthology.orchestra.run.smithi059:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-26T22:06:44.069 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T22:06:44.069 DEBUG:teuthology.orchestra.run.smithi076:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-12-26T22:06:44.100 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T22:06:44.101 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-12-26T22:06:44.110 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-12-26T22:06:44.152 INFO:teuthology.task.internal:Duration was 1382.331071 seconds 2023-12-26T22:06:44.152 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-12-26T22:06:44.194 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-12-26T22:06:44.195 DEBUG:teuthology.orchestra.run.smithi059:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-26T22:06:44.197 DEBUG:teuthology.orchestra.run.smithi076:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-12-26T22:06:44.243 INFO:teuthology.orchestra.run.smithi076.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-26T22:06:44.245 INFO:teuthology.orchestra.run.smithi059.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-12-26T22:06:44.559 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-12-26T22:06:44.560 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi059.front.sepia.ceph.com 2023-12-26T22:06:44.560 DEBUG:teuthology.orchestra.run.smithi059:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-26T22:06:44.586 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi076.front.sepia.ceph.com 2023-12-26T22:06:44.586 DEBUG:teuthology.orchestra.run.smithi076:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-12-26T22:06:44.608 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-12-26T22:06:44.608 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-26T22:06:44.628 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-12-26T22:06:44.693 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-12-26T22:06:44.693 DEBUG:teuthology.orchestra.run.smithi059:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-26T22:06:44.695 DEBUG:teuthology.orchestra.run.smithi076:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-12-26T22:06:44.781 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-12-26T22:06:44.791 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-12-26T22:06:44.791 DEBUG:teuthology.orchestra.run.smithi059:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-26T22:06:44.819 DEBUG:teuthology.orchestra.run.smithi076:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-12-26T22:06:44.846 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-12-26T22:06:44.858 DEBUG:teuthology.orchestra.run.smithi059:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-26T22:06:44.862 DEBUG:teuthology.orchestra.run.smithi076:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-12-26T22:06:44.887 INFO:teuthology.orchestra.run.smithi059.stdout:kernel.core_pattern = core 2023-12-26T22:06:44.912 INFO:teuthology.orchestra.run.smithi076.stdout:kernel.core_pattern = core 2023-12-26T22:06:44.930 DEBUG:teuthology.orchestra.run.smithi059:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-26T22:06:44.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T22:06:44.975 DEBUG:teuthology.orchestra.run.smithi076:> test -e /home/ubuntu/cephtest/archive/coredump 2023-12-26T22:06:44.991 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-12-26T22:06:44.992 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-12-26T22:06:45.002 INFO:teuthology.task.internal:Transferring archived files... 2023-12-26T22:06:45.003 DEBUG:teuthology.misc:Transferring archived files from smithi059:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/remote/smithi059 2023-12-26T22:06:45.003 DEBUG:teuthology.orchestra.run.smithi059:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-26T22:06:45.055 DEBUG:teuthology.misc:Transferring archived files from smithi076:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828/remote/smithi076 2023-12-26T22:06:45.055 DEBUG:teuthology.orchestra.run.smithi076:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-12-26T22:06:45.093 INFO:teuthology.task.internal:Removing archive directory... 2023-12-26T22:06:45.094 DEBUG:teuthology.orchestra.run.smithi059:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-26T22:06:45.096 DEBUG:teuthology.orchestra.run.smithi076:> rm -rf -- /home/ubuntu/cephtest/archive 2023-12-26T22:06:45.145 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-12-26T22:06:45.155 INFO:teuthology.task.internal:Not uploading archives. 2023-12-26T22:06:45.155 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-12-26T22:06:45.178 INFO:teuthology.task.internal:Tidying up after the test... 2023-12-26T22:06:45.178 DEBUG:teuthology.orchestra.run.smithi059:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-26T22:06:45.181 DEBUG:teuthology.orchestra.run.smithi076:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-12-26T22:06:45.196 INFO:teuthology.orchestra.run.smithi059.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 26 22:06 /home/ubuntu/cephtest 2023-12-26T22:06:45.203 INFO:teuthology.orchestra.run.smithi076.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Dec 26 22:06 /home/ubuntu/cephtest 2023-12-26T22:06:45.204 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-12-26T22:06:45.220 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-12-26T22:06:45.303 INFO:teuthology.nuke:Checking targets against current locks 2023-12-26T22:06:45.330 DEBUG:teuthology.nuke:shortname: smithi059 2023-12-26T22:06:45.331 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-26T22:06:45.351 DEBUG:teuthology.nuke:shortname: smithi076 2023-12-26T22:06:45.351 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-12-26T22:06:45.355 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi059.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-26 21:34:28.143534', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBOGAaTChfsyvSRP//8gkxCjUiJUT0U8hCULLVuGEMXUhbKRjMcXGWXuO95EFEdreiWwZxip00+jDqu5vPRYrcSA='} 2023-12-26T22:06:45.368 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi076.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-12-26_19:48:51-fs-wip-yuri-testing-2023-12-26-0957-quincy-distro-default-smithi/7501828', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-12-26 21:34:28.140704', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJyxUj0593trmMuHzbKoaJL4pP7z9BfSI3SXn3ElkIWWNXbeqeVa+nPHdl4PRAe1dazFO/n+gfMpmPJeQcJ/4Xo='} 2023-12-26T22:06:45.404 INFO:teuthology.orchestra.console.smithi059:Power off 2023-12-26T22:06:45.404 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-26T22:06:45.420 INFO:teuthology.orchestra.console.smithi076:Power off 2023-12-26T22:06:45.420 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-12-26T22:06:45.429 DEBUG:teuthology.orchestra.console.smithi059:power off output: Chassis Power Control: Down/Off 2023-12-26T22:06:45.429 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:45.442 DEBUG:teuthology.orchestra.console.smithi076:power off output: Chassis Power Control: Down/Off 2023-12-26T22:06:45.442 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:45.450 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-12-26T22:06:45.465 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-26T22:06:49.452 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:49.466 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:49.562 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-12-26T22:06:49.575 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-26T22:06:53.565 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:53.576 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:53.676 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is on 2023-12-26T22:06:53.685 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is on 2023-12-26T22:06:57.682 DEBUG:teuthology.orchestra.console.smithi059:pexpect command: ipmitool -H smithi059.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:57.690 DEBUG:teuthology.orchestra.console.smithi076:pexpect command: ipmitool -H smithi076.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-12-26T22:06:57.792 DEBUG:teuthology.orchestra.console.smithi059:check power output: Chassis Power is off 2023-12-26T22:06:57.799 DEBUG:teuthology.orchestra.console.smithi076:check power output: Chassis Power is off 2023-12-26T22:06:57.894 INFO:teuthology.orchestra.console.smithi059:Power off completed 2023-12-26T22:06:57.900 INFO:teuthology.orchestra.console.smithi076:Power off completed 2023-12-26T22:06:58.005 INFO:teuthology.run:Summary data: description: fs/upgrade/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/pacific 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1382.3310713768005 failure_reason: 'Command failed on smithi076 with status 5: ''sudo systemctl stop ceph-761f1ff0-a439-11ee-95a5-87774f69a715@mon.smithi076''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=a673131c2a824828b9d03caf51701a00 status: fail success: false 2023-12-26T22:06:58.006 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-12-26T22:06:58.088 INFO:teuthology.run:FAIL