2024-06-15T00:36:50.645 INFO:root:teuthology version: 0.0.1.dev316+gc1b06c2 2024-06-15T00:36:50.645 DEBUG:teuthology.run:Teuthology command: teuthology --owner scheduled_yuriw@teuthology --name yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi --description orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} --verbose --archive /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127 -- /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/orig.config.yaml 2024-06-15T00:36:50.691 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T00:36:50.782 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127 branch: wip-yuri5-testing-2024-06-08-0902-reef description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7756127' kernel: kdb: 1 sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/quincy ' name: yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi no_nested_subset: false os_type: centos os_version: 9.stream overrides: admin_socket: branch: wip-yuri5-testing-2024-06-08-0902-reef ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - FS_DEGRADED - FS_INLINE_DATA_DEPRECATED - FS_WITH_FAILED_MDS - MDS_ALL_DOWN - MDS_DAMAGE - MDS_DEGRADED - MDS_FAILED - MDS_INSUFFICIENT_STANDBY - MDS_UP_LESS_THAN_MAX - POOL_APP_NOT_ENABLED - overall HEALTH_ - Replacing daemon - OSD_DOWN - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc kclient: syntax: v1 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-reef-drop-centos8 sha1: e3daf20513a97cf7004cd9e7bfdba8f5a015acbc owner: scheduled_yuriw@teuthology priority: 71 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 302 sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc sleep_before_teardown: 0 subset: 111/120000 suite: orch suite_branch: wip-reef-drop-centos8 suite_path: /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa suite_relpath: qa suite_repo: https://github.com/cbodley/ceph.git suite_sha1: e3daf20513a97cf7004cd9e7bfdba8f5a015acbc targets: smithi105.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFPlVhQf5eGr56wfYonetd2LMELJ4UOS5T/9LCoIc4ONKdh+5RZeDaCpI1Rj76SZ+io9YeFv7wERmOc1WW+fhdQ= smithi143.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEYsGe+pgwO+kWQpJ5lpQptjzbgLh1uUVhnVgE4V5duW/QjFJg4u2Rv9d8novCTL66Qw8SHxXxv2DqOa5IO+V00= tasks: - install: branch: quincy exclude_packages: - ceph-volume - print: '**** done install task...' - cephadm: cephadm_branch: quincy cephadm_git_url: https://github.com/ceph/ceph conf: osd: osd_class_default_list: '*' osd_class_load_list: '*' image: quay.ceph.io/ceph-ci/ceph:quincy roleless: true - print: '**** done end installing quincy cephadm ...' - cephadm.shell: host.a: - ceph config set mgr mgr/cephadm/use_repo_digest true --force - print: '**** done cephadm.shell ceph config set mgr...' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: - /cephfs/home/yuriw/src/github.com_cbodley_ceph_wip-reef-drop-centos8/qa/suites/orch/cephadm/mds_upgrade_sequence/tasks/3-upgrade-mgr-staggered.yaml meta: {} postmerge: [] variables: fail_fs: false teuthology_branch: main teuthology_sha1: c1b06c2b5d3a02a3583265a115f09e052855bcf2 timestamp: 2024-06-14_18:29:32 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mgr mgr/orchestrator/fail_fs false || true - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; ceph health detail ; sleep 30 ; done - ceph orch ps - ceph orch upgrade status - ceph health detail - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2698841 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-06-15T00:36:50.783 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa; will attempt to use it 2024-06-15T00:36:50.783 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks 2024-06-15T00:36:50.783 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-06-15T00:36:50.785 INFO:teuthology.task.internal:Checking packages... 2024-06-15T00:36:50.803 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '6f4b09560ed399c3f139402a02f389c2fc8bb2cc' 2024-06-15T00:36:50.804 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:36:50.804 INFO:teuthology.packaging:ref: None 2024-06-15T00:36:50.804 INFO:teuthology.packaging:tag: None 2024-06-15T00:36:50.804 INFO:teuthology.packaging:branch: wip-yuri5-testing-2024-06-08-0902-reef 2024-06-15T00:36:50.804 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:36:50.804 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=wip-yuri5-testing-2024-06-08-0902-reef 2024-06-15T00:36:50.953 INFO:teuthology.task.internal:Found packages for ceph version 18.2.2-1443.g6f4b0956 2024-06-15T00:36:50.955 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-06-15T00:36:50.961 INFO:teuthology.task.internal:no buildpackages task found 2024-06-15T00:36:50.961 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-06-15T00:36:51.011 INFO:teuthology.task.internal:Saving configuration 2024-06-15T00:36:51.030 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-06-15T00:36:51.043 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-06-15T00:36:51.068 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi105.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-15 00:31:53.902866', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBFPlVhQf5eGr56wfYonetd2LMELJ4UOS5T/9LCoIc4ONKdh+5RZeDaCpI1Rj76SZ+io9YeFv7wERmOc1WW+fhdQ='} 2024-06-15T00:36:51.092 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi143.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '9.stream', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-06-15 00:31:53.904750', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBEYsGe+pgwO+kWQpJ5lpQptjzbgLh1uUVhnVgE4V5duW/QjFJg4u2Rv9d8novCTL66Qw8SHxXxv2DqOa5IO+V00='} 2024-06-15T00:36:51.092 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-06-15T00:36:51.098 INFO:teuthology.task.internal:roles: ubuntu@smithi105.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-06-15T00:36:51.098 INFO:teuthology.task.internal:roles: ubuntu@smithi143.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-06-15T00:36:51.098 INFO:teuthology.run_tasks:Running task console_log... 2024-06-15T00:36:51.171 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f4dcaa93be0>, signals=[15]) 2024-06-15T00:36:51.172 INFO:teuthology.run_tasks:Running task internal.connect... 2024-06-15T00:36:51.177 INFO:teuthology.task.internal:Opening connections... 2024-06-15T00:36:51.177 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi105.front.sepia.ceph.com 2024-06-15T00:36:51.178 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:36:51.255 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi143.front.sepia.ceph.com 2024-06-15T00:36:51.256 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:36:51.331 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-06-15T00:36:51.338 DEBUG:teuthology.orchestra.run.smithi105:> uname -m 2024-06-15T00:36:51.355 INFO:teuthology.orchestra.run.smithi105.stdout:x86_64 2024-06-15T00:36:51.356 DEBUG:teuthology.orchestra.run.smithi105:> cat /etc/os-release 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:NAME="CentOS Stream" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION="9" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:ID="centos" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:ID_LIKE="rhel fedora" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:VERSION_ID="9" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:PLATFORM_ID="platform:el9" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:ANSI_COLOR="0;31" 2024-06-15T00:36:51.413 INFO:teuthology.orchestra.run.smithi105.stdout:LOGO="fedora-logo-icon" 2024-06-15T00:36:51.414 INFO:teuthology.orchestra.run.smithi105.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-15T00:36:51.414 INFO:teuthology.orchestra.run.smithi105.stdout:HOME_URL="https://centos.org/" 2024-06-15T00:36:51.414 INFO:teuthology.orchestra.run.smithi105.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-15T00:36:51.414 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-15T00:36:51.414 INFO:teuthology.orchestra.run.smithi105.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-15T00:36:51.415 INFO:teuthology.lock.ops:Updating smithi105.front.sepia.ceph.com on lock server 2024-06-15T00:36:51.438 DEBUG:teuthology.orchestra.run.smithi143:> uname -m 2024-06-15T00:36:51.455 INFO:teuthology.orchestra.run.smithi143.stdout:x86_64 2024-06-15T00:36:51.456 DEBUG:teuthology.orchestra.run.smithi143:> cat /etc/os-release 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:NAME="CentOS Stream" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION="9" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:ID="centos" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:ID_LIKE="rhel fedora" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:VERSION_ID="9" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:PLATFORM_ID="platform:el9" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:PRETTY_NAME="CentOS Stream 9" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:ANSI_COLOR="0;31" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:LOGO="fedora-logo-icon" 2024-06-15T00:36:51.513 INFO:teuthology.orchestra.run.smithi143.stdout:CPE_NAME="cpe:/o:centos:centos:9" 2024-06-15T00:36:51.514 INFO:teuthology.orchestra.run.smithi143.stdout:HOME_URL="https://centos.org/" 2024-06-15T00:36:51.514 INFO:teuthology.orchestra.run.smithi143.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-06-15T00:36:51.514 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 9" 2024-06-15T00:36:51.514 INFO:teuthology.orchestra.run.smithi143.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-06-15T00:36:51.514 INFO:teuthology.lock.ops:Updating smithi143.front.sepia.ceph.com on lock server 2024-06-15T00:36:51.538 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-06-15T00:36:51.547 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-06-15T00:36:51.588 INFO:teuthology.task.internal:Checking for old test directory... 2024-06-15T00:36:51.589 DEBUG:teuthology.orchestra.run.smithi105:> test '!' -e /home/ubuntu/cephtest 2024-06-15T00:36:51.591 DEBUG:teuthology.orchestra.run.smithi143:> test '!' -e /home/ubuntu/cephtest 2024-06-15T00:36:51.607 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-06-15T00:36:51.613 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-06-15T00:36:51.613 DEBUG:teuthology.orchestra.run.smithi105:> test -z $(ls -A /var/lib/ceph) 2024-06-15T00:36:51.648 DEBUG:teuthology.orchestra.run.smithi143:> test -z $(ls -A /var/lib/ceph) 2024-06-15T00:36:51.691 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-06-15T00:36:51.748 INFO:teuthology.run_tasks:Running task kernel... 2024-06-15T00:36:51.763 INFO:teuthology.task.kernel:normalize config orig: {'kdb': 1, 'sha1': 'distro'} 2024-06-15T00:36:51.763 INFO:teuthology.task.kernel:config {'host.a': {'kdb': 1, 'sha1': 'distro'}, 'host.b': {'kdb': 1, 'sha1': 'distro'}}, timeout 300 2024-06-15T00:36:51.763 DEBUG:teuthology.orchestra.run.smithi105:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-15T00:36:51.764 DEBUG:teuthology.orchestra.run.smithi143:> test -f /run/.containerenv -o -f /.dockerenv 2024-06-15T00:36:51.781 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:36:51.782 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-06-15T00:36:51.783 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:36:51.783 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-06-15T00:36:51.840 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-437.el9.x86_64 2024-06-15T00:36:51.840 INFO:teuthology.task.kernel:Running kernel on smithi105: 5.14.0-437.el9.x86_64 2024-06-15T00:36:51.840 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum install -y kernel 2024-06-15T00:36:51.841 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-437.el9.x86_64 2024-06-15T00:36:51.841 INFO:teuthology.task.kernel:Running kernel on smithi143: 5.14.0-437.el9.x86_64 2024-06-15T00:36:51.842 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum install -y kernel 2024-06-15T00:36:54.283 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 11 MB/s | 8.1 MB 00:00 2024-06-15T00:36:58.036 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 1.8 MB/s | 8.1 MB 00:04 2024-06-15T00:37:00.217 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 4.1 MB/s | 20 MB 00:04 2024-06-15T00:37:04.037 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 4.0 MB/s | 20 MB 00:04 2024-06-15T00:37:04.317 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 16 MB/s | 6.3 MB 00:00 2024-06-15T00:37:06.143 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 17 kB 00:00 2024-06-15T00:37:07.530 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 17 MB/s | 21 MB 00:01 2024-06-15T00:37:09.749 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 3.4 MB/s | 6.3 MB 00:01 2024-06-15T00:37:11.454 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 33 kB/s | 17 kB 00:00 2024-06-15T00:37:12.409 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 26 MB/s | 21 MB 00:00 2024-06-15T00:37:12.844 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 20 kB/s | 1.7 kB 00:00 2024-06-15T00:37:14.859 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-15T00:37:14.859 INFO:teuthology.orchestra.run.smithi105.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-15T00:37:14.951 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:37:14.962 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:37:14.962 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-06-15T00:37:14.962 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:37:14.962 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:37:14.962 INFO:teuthology.orchestra.run.smithi105.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-06-15T00:37:14.963 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:37:14.964 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 81 M 2024-06-15T00:37:14.964 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 124 M 2024-06-15T00:37:14.964 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:37:15.363 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 4.0 MB/s | 989 kB 00:00 2024-06-15T00:37:17.079 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): kernel-modules-core-5.14.0-457.el9.x86_6 17 MB/s | 29 MB 00:01 2024-06-15T00:37:17.263 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): kernel-core-5.14.0-457.el9.x86_64.rpm 7.5 MB/s | 16 MB 00:02 2024-06-15T00:37:17.681 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 30 kB/s | 1.7 kB 00:00 2024-06-15T00:37:18.406 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 11 MB/s | 35 MB 00:03 2024-06-15T00:37:18.406 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:37:18.406 INFO:teuthology.orchestra.run.smithi105.stdout:Total 23 MB/s | 81 MB 00:03 2024-06-15T00:37:19.108 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:37:19.447 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:37:19.447 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:37:19.738 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-239.el9.x86_64 is already installed. 2024-06-15T00:37:19.738 INFO:teuthology.orchestra.run.smithi143.stdout:Package kernel-5.14.0-437.el9.x86_64 is already installed. 2024-06-15T00:37:19.842 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout: kernel x86_64 5.14.0-457.el9 baseos 989 k 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core x86_64 5.14.0-457.el9 baseos 16 M 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules x86_64 5.14.0-457.el9 baseos 35 M 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core x86_64 5.14.0-457.el9 baseos 29 M 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:37:19.855 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:37:19.856 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-06-15T00:37:19.856 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:37:19.856 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 81 M 2024-06-15T00:37:19.856 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 124 M 2024-06-15T00:37:19.856 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:37:20.083 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): kernel-5.14.0-457.el9.x86_64.rpm 6.0 MB/s | 989 kB 00:00 2024-06-15T00:37:20.358 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): kernel-core-5.14.0-457.el9.x86_64.rpm 37 MB/s | 16 MB 00:00 2024-06-15T00:37:20.387 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:37:20.387 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:37:20.733 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): kernel-modules-core-5.14.0-457.el9.x86_6 45 MB/s | 29 MB 00:00 2024-06-15T00:37:21.050 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): kernel-modules-5.14.0-457.el9.x86_64.rpm 31 MB/s | 35 MB 00:01 2024-06-15T00:37:21.051 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:37:21.051 INFO:teuthology.orchestra.run.smithi143.stdout:Total 68 MB/s | 81 MB 00:01 2024-06-15T00:37:21.759 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:37:22.119 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:37:22.119 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:37:22.795 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:37:23.083 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:37:23.084 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:37:23.499 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-15T00:37:23.597 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-15T00:37:25.494 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:37:26.257 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-15T00:37:26.393 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-core-5.14.0-457.el9.x86_64 1/4 2024-06-15T00:37:26.398 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-15T00:37:26.486 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-15T00:37:29.183 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-15T00:37:29.347 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-15T00:37:31.139 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-15T00:37:31.254 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:37:34.089 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-15T00:37:34.437 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : kernel-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:37:36.035 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:37:39.172 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:05.327 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:05.369 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:05.756 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:05.756 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-15T00:38:05.756 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-15T00:38:05.756 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-15T00:38:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:38:06.582 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:38:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-15T00:38:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-15T00:38:06.582 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-15T00:38:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-15T00:38:06.583 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:38:06.583 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:38:06.911 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-06-15T00:38:06.981 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-15T00:38:06.981 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-15T00:38:06.982 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-15T00:38:06.982 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-15T00:38:06.982 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-15T00:38:06.982 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-06-15T00:38:06.982 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi105.front.sepia.ceph.com, path=None, version=distro) 2024-06-15T00:38:06.982 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q kernel | sort -rV | head -n 1 2024-06-15T00:38:07.035 INFO:teuthology.orchestra.run.smithi105.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-15T00:38:07.035 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-15T00:38:07.035 DEBUG:teuthology.orchestra.run.smithi105:> sudo rpm -qi grub2-tools 2024-06-15T00:38:07.070 INFO:teuthology.orchestra.run.smithi105.stdout:Name : grub2-tools 2024-06-15T00:38:07.070 INFO:teuthology.orchestra.run.smithi105.stdout:Epoch : 1 2024-06-15T00:38:07.070 INFO:teuthology.orchestra.run.smithi105.stdout:Version : 2.06 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Release : 68.el9 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Architecture: x86_64 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Group : Unspecified 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Size : 8264254 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:License : GPLv3+ 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-15T00:38:07.071 INFO:teuthology.orchestra.run.smithi105.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:Packager : builder@centos.org 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:Vendor : CentOS 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:Summary : Support tools for GRUB. 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:Description : 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:hardware devices. 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:38:07.072 INFO:teuthology.orchestra.run.smithi105.stdout:This subpackage provides tools for support of all platforms. 2024-06-15T00:38:07.073 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-15T00:38:07.073 INFO:teuthology.task.kernel:Updating grub on smithi105 to boot 5.14.0-457.el9.x86_64 2024-06-15T00:38:07.074 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-15T00:38:07.679 INFO:teuthology.orchestra.run.smithi105.stderr:Generating grub configuration file ... 2024-06-15T00:38:08.658 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-core-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:08.698 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-modules-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:08.975 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: kernel-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:08.975 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-5.14.0-457.el9.x86_64 1/4 2024-06-15T00:38:08.975 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-core-5.14.0-457.el9.x86_64 2/4 2024-06-15T00:38:08.975 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-5.14.0-457.el9.x86_64 3/4 2024-06-15T00:38:09.129 INFO:teuthology.orchestra.run.smithi105.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-15T00:38:09.158 INFO:teuthology.orchestra.run.smithi105.stderr:done 2024-06-15T00:38:09.160 DEBUG:teuthology.orchestra.run.smithi105:> mktemp 2024-06-15T00:38:09.176 INFO:teuthology.orchestra.run.smithi105.stdout:/tmp/tmp.zQjdsxYKfG 2024-06-15T00:38:09.176 DEBUG:teuthology.orchestra.run.smithi105:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.zQjdsxYKfG 2024-06-15T00:38:09.242 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0666 /tmp/tmp.zQjdsxYKfG 2024-06-15T00:38:09.376 DEBUG:teuthology.orchestra.remote:smithi105:/tmp/tmp.zQjdsxYKfG is 6KB 2024-06-15T00:38:09.424 DEBUG:teuthology.orchestra.run.smithi105:> rm -fr /tmp/tmp.zQjdsxYKfG 2024-06-15T00:38:09.438 DEBUG:teuthology.orchestra.run.smithi105:> sudo /bin/ls /boot/loader/entries || true 2024-06-15T00:38:09.499 INFO:teuthology.orchestra.run.smithi105.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-15T00:38:09.499 INFO:teuthology.orchestra.run.smithi105.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-15T00:38:09.499 INFO:teuthology.orchestra.run.smithi105.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-15T00:38:09.499 INFO:teuthology.orchestra.run.smithi105.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-15T00:38:09.500 DEBUG:teuthology.orchestra.run.smithi105:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-15T00:38:09.647 DEBUG:teuthology.orchestra.run.smithi105:> sudo shutdown -r now 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : kernel-modules-core-5.14.0-457.el9.x86_64 4/4 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-5.14.0-457.el9.x86_64 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-core-5.14.0-457.el9.x86_64 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-5.14.0-457.el9.x86_64 2024-06-15T00:38:09.773 INFO:teuthology.orchestra.run.smithi143.stdout: kernel-modules-core-5.14.0-457.el9.x86_64 2024-06-15T00:38:09.774 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:38:09.774 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:38:10.127 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-06-15T00:38:10.223 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-15T00:38:10.223 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-15T00:38:10.223 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 5.14.0-437.el9.x86_64 Expected: 5.14.0-457.el9.x86_64 2024-06-15T00:38:10.223 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-06-15T00:38:10.223 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-06-15T00:38:10.223 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-06-15T00:38:10.223 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi143.front.sepia.ceph.com, path=None, version=distro) 2024-06-15T00:38:10.223 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q kernel | sort -rV | head -n 1 2024-06-15T00:38:10.275 INFO:teuthology.orchestra.run.smithi143.stdout:kernel-5.14.0-457.el9.x86_64 2024-06-15T00:38:10.276 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 5.14.0-457.el9.x86_64 2024-06-15T00:38:10.276 DEBUG:teuthology.orchestra.run.smithi143:> sudo rpm -qi grub2-tools 2024-06-15T00:38:10.306 INFO:teuthology.orchestra.run.smithi143.stdout:Name : grub2-tools 2024-06-15T00:38:10.306 INFO:teuthology.orchestra.run.smithi143.stdout:Epoch : 1 2024-06-15T00:38:10.306 INFO:teuthology.orchestra.run.smithi143.stdout:Version : 2.06 2024-06-15T00:38:10.306 INFO:teuthology.orchestra.run.smithi143.stdout:Release : 68.el9 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Architecture: x86_64 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Install Date: Fri 19 Apr 2024 09:32:39 PM UTC 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Group : Unspecified 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Size : 8264254 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:License : GPLv3+ 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Signature : RSA/SHA256, Mon 07 Aug 2023 01:57:16 PM UTC, Key ID 05b555b38483c65d 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Source RPM : grub2-2.06-68.el9.src.rpm 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Build Date : Wed 02 Aug 2023 03:15:48 PM UTC 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Build Host : x86-05.stream.rdu2.redhat.com 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Packager : builder@centos.org 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Vendor : CentOS 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:URL : http://www.gnu.org/software/grub/ 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Summary : Support tools for GRUB. 2024-06-15T00:38:10.307 INFO:teuthology.orchestra.run.smithi143.stdout:Description : 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:customizable bootloader with modular architecture. It supports a rich 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:variety of kernel formats, file systems, computer architectures and 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:hardware devices. 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:38:10.308 INFO:teuthology.orchestra.run.smithi143.stdout:This subpackage provides tools for support of all platforms. 2024-06-15T00:38:10.309 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-06-15T00:38:10.309 INFO:teuthology.task.kernel:Updating grub on smithi143 to boot 5.14.0-457.el9.x86_64 2024-06-15T00:38:10.309 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-06-15T00:38:10.944 INFO:teuthology.orchestra.run.smithi143.stderr:Generating grub configuration file ... 2024-06-15T00:38:12.333 INFO:teuthology.orchestra.run.smithi143.stderr:Adding boot menu entry for UEFI Firmware Settings ... 2024-06-15T00:38:12.363 INFO:teuthology.orchestra.run.smithi143.stderr:done 2024-06-15T00:38:12.365 DEBUG:teuthology.orchestra.run.smithi143:> mktemp 2024-06-15T00:38:12.383 INFO:teuthology.orchestra.run.smithi143.stdout:/tmp/tmp.iyf2IWDQ66 2024-06-15T00:38:12.383 DEBUG:teuthology.orchestra.run.smithi143:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.iyf2IWDQ66 2024-06-15T00:38:12.448 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0666 /tmp/tmp.iyf2IWDQ66 2024-06-15T00:38:12.590 DEBUG:teuthology.orchestra.remote:smithi143:/tmp/tmp.iyf2IWDQ66 is 6KB 2024-06-15T00:38:12.638 DEBUG:teuthology.orchestra.run.smithi143:> rm -fr /tmp/tmp.iyf2IWDQ66 2024-06-15T00:38:12.652 DEBUG:teuthology.orchestra.run.smithi143:> sudo /bin/ls /boot/loader/entries || true 2024-06-15T00:38:12.714 INFO:teuthology.orchestra.run.smithi143.stdout:7fc8828e06d0482694bcf4a447d13cb6-0-rescue.conf 2024-06-15T00:38:12.714 INFO:teuthology.orchestra.run.smithi143.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-239.el9.x86_64.conf 2024-06-15T00:38:12.714 INFO:teuthology.orchestra.run.smithi143.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-437.el9.x86_64.conf 2024-06-15T00:38:12.714 INFO:teuthology.orchestra.run.smithi143.stdout:7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64.conf 2024-06-15T00:38:12.715 DEBUG:teuthology.orchestra.run.smithi143:> sudo grub2-set-default 7fc8828e06d0482694bcf4a447d13cb6-5.14.0-457.el9.x86_64 2024-06-15T00:38:12.866 DEBUG:teuthology.orchestra.run.smithi143:> sudo shutdown -r now 2024-06-15T00:38:39.677 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-06-15T00:38:39.678 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:38:39.679 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:38:42.872 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-06-15T00:38:42.872 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:38:42.872 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:38:58.079 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-06-15T00:39:01.406 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-06-15T00:39:07.082 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:39:07.083 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:39:10.142 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-06-15T00:39:10.407 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:39:10.408 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:39:13.470 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-06-15T00:39:22.151 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:39:22.152 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:39:25.472 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:39:25.473 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:39:40.574 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.105 2024-06-15T00:39:43.902 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.143 2024-06-15T00:39:55.584 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:39:55.585 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:39:56.015 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-06-15T00:39:56.416 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:39:56.416 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "5.14.0-457.el9.x86_64"... 2024-06-15T00:39:56.417 DEBUG:teuthology.orchestra.run.smithi105:> uname -r 2024-06-15T00:39:56.431 INFO:teuthology.orchestra.run.smithi105.stdout:5.14.0-457.el9.x86_64 2024-06-15T00:39:56.431 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-15T00:39:56.431 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-15T00:39:56.431 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-15T00:39:57.432 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-06-15T00:39:57.432 DEBUG:teuthology.orchestra.run.smithi105:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-15T00:39:57.509 INFO:teuthology.orchestra.run.smithi105.stdout:ttyS1 2024-06-15T00:39:57.540 DEBUG:teuthology.parallel:result is None 2024-06-15T00:39:58.903 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:39:58.903 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:39:59.309 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-06-15T00:39:59.996 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:39:59.996 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "5.14.0-457.el9.x86_64"... 2024-06-15T00:39:59.997 DEBUG:teuthology.orchestra.run.smithi143:> uname -r 2024-06-15T00:40:00.042 INFO:teuthology.orchestra.run.smithi143.stdout:5.14.0-457.el9.x86_64 2024-06-15T00:40:00.042 DEBUG:teuthology.task.kernel:current kernel version is 5.14.0-457.el9.x86_64 vs 5.14.0-457.el9.x86_64 2024-06-15T00:40:00.042 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-06-15T00:40:00.042 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-06-15T00:40:01.042 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-06-15T00:40:01.043 DEBUG:teuthology.orchestra.run.smithi143:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-06-15T00:40:01.152 INFO:teuthology.orchestra.run.smithi143.stdout:ttyS1 2024-06-15T00:40:01.211 DEBUG:teuthology.parallel:result is None 2024-06-15T00:40:01.211 INFO:teuthology.run_tasks:Running task internal.base... 2024-06-15T00:40:01.218 INFO:teuthology.task.internal:Creating test directory... 2024-06-15T00:40:01.218 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-15T00:40:01.221 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-06-15T00:40:01.239 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-06-15T00:40:01.254 INFO:teuthology.run_tasks:Running task internal.archive... 2024-06-15T00:40:01.288 INFO:teuthology.task.internal:Creating archive directory... 2024-06-15T00:40:01.288 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-15T00:40:01.290 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-06-15T00:40:01.323 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-06-15T00:40:01.329 INFO:teuthology.task.internal:Enabling coredump saving... 2024-06-15T00:40:01.329 DEBUG:teuthology.orchestra.run.smithi105:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-15T00:40:01.357 DEBUG:teuthology.orchestra.run.smithi143:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-06-15T00:40:01.380 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T00:40:01.390 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T00:40:01.443 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T00:40:01.453 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-06-15T00:40:01.455 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-06-15T00:40:01.464 INFO:teuthology.task.internal:Configuring sudo... 2024-06-15T00:40:01.464 DEBUG:teuthology.orchestra.run.smithi105:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-15T00:40:01.466 DEBUG:teuthology.orchestra.run.smithi143:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-06-15T00:40:01.491 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-06-15T00:40:01.499 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-06-15T00:40:01.500 DEBUG:teuthology.orchestra.run.smithi105:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-15T00:40:01.532 DEBUG:teuthology.orchestra.run.smithi143:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-06-15T00:40:01.546 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T00:40:01.628 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T00:40:01.671 DEBUG:teuthology.orchestra.run.smithi105:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T00:40:01.753 DEBUG:teuthology.orchestra.run.smithi105:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T00:40:01.794 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:40:01.794 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-15T00:40:01.860 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T00:40:01.902 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-06-15T00:40:01.952 DEBUG:teuthology.orchestra.run.smithi143:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T00:40:02.036 DEBUG:teuthology.orchestra.run.smithi143:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-06-15T00:40:02.075 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:40:02.075 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-06-15T00:40:02.139 DEBUG:teuthology.orchestra.run.smithi105:> sudo service rsyslog restart 2024-06-15T00:40:02.142 DEBUG:teuthology.orchestra.run.smithi143:> sudo service rsyslog restart 2024-06-15T00:40:02.195 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-15T00:40:02.234 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-15T00:40:02.736 INFO:teuthology.run_tasks:Running task internal.timer... 2024-06-15T00:40:02.744 INFO:teuthology.task.internal:Starting timer... 2024-06-15T00:40:02.744 INFO:teuthology.run_tasks:Running task pcp... 2024-06-15T00:40:02.784 INFO:teuthology.run_tasks:Running task selinux... 2024-06-15T00:40:02.858 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-06-15T00:40:02.858 DEBUG:teuthology.orchestra.run.smithi105:> sudo service auditd rotate 2024-06-15T00:40:02.942 INFO:teuthology.orchestra.run.smithi105.stdout:Rotating logs: 2024-06-15T00:40:02.944 DEBUG:teuthology.orchestra.run.smithi143:> sudo service auditd rotate 2024-06-15T00:40:03.010 INFO:teuthology.orchestra.run.smithi143.stdout:Rotating logs: 2024-06-15T00:40:03.012 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-06-15T00:40:03.012 DEBUG:teuthology.orchestra.run.smithi105:> /usr/sbin/getenforce 2024-06-15T00:40:03.060 INFO:teuthology.orchestra.run.smithi105.stdout:Permissive 2024-06-15T00:40:03.060 DEBUG:teuthology.orchestra.run.smithi143:> /usr/sbin/getenforce 2024-06-15T00:40:03.090 INFO:teuthology.orchestra.run.smithi143.stdout:Permissive 2024-06-15T00:40:03.091 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi105.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi143.front.sepia.ceph.com': 'permissive'} 2024-06-15T00:40:03.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-15T00:40:03.117 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:40:03.117 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-15T00:40:03.156 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:40:03.156 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-06-15T00:40:03.157 DEBUG:teuthology.orchestra.run.smithi105:> sudo /usr/sbin/setenforce permissive 2024-06-15T00:40:03.182 DEBUG:teuthology.orchestra.run.smithi143:> sudo /usr/sbin/setenforce permissive 2024-06-15T00:40:03.230 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-06-15T00:40:03.238 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-06-15T00:40:03.248 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-06-15T00:40:03.250 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi105.front.sepia.ceph.com,smithi143.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-06-15T00:44:23.741 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi105.front.sepia.ceph.com'), Remote(name='ubuntu@smithi143.front.sepia.ceph.com')] 2024-06-15T00:44:23.743 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:44:23.744 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi105.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:44:23.822 DEBUG:teuthology.orchestra.run.smithi105:> true 2024-06-15T00:44:23.899 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi105.front.sepia.ceph.com' 2024-06-15T00:44:23.899 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:44:23.900 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi143.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-06-15T00:44:23.974 DEBUG:teuthology.orchestra.run.smithi143:> true 2024-06-15T00:44:24.050 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi143.front.sepia.ceph.com' 2024-06-15T00:44:24.050 INFO:teuthology.run_tasks:Running task clock... 2024-06-15T00:44:24.063 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-06-15T00:44:24.063 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-15T00:44:24.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T00:44:24.065 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-06-15T00:44:24.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T00:44:24.093 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-15T00:44:24.107 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-15T00:44:24.119 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-06-15T00:44:24.131 INFO:teuthology.orchestra.run.smithi105.stderr:sudo: ntpd: command not found 2024-06-15T00:44:24.133 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-06-15T00:44:24.142 INFO:teuthology.orchestra.run.smithi105.stdout:506 Cannot talk to daemon 2024-06-15T00:44:24.156 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-15T00:44:24.157 INFO:teuthology.orchestra.run.smithi143.stderr:sudo: ntpd: command not found 2024-06-15T00:44:24.168 INFO:teuthology.orchestra.run.smithi143.stdout:506 Cannot talk to daemon 2024-06-15T00:44:24.170 INFO:teuthology.orchestra.run.smithi105.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-15T00:44:24.182 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-06-15T00:44:24.196 INFO:teuthology.orchestra.run.smithi143.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-06-15T00:44:24.229 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-06-15T00:44:24.233 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-15T00:44:24.233 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-06-15T00:44:24.233 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.233 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.233 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.233 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.249 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-06-15T00:44:24.253 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-15T00:44:24.253 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-06-15T00:44:24.253 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.253 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.253 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.253 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:44:24.254 INFO:teuthology.run_tasks:Running task install... 2024-06-15T00:44:24.262 DEBUG:teuthology.task.install:project ceph 2024-06-15T00:44:24.262 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'}} 2024-06-15T00:44:24.262 DEBUG:teuthology.task.install:config {'branch': 'quincy', 'exclude_packages': ['ceph-volume'], 'flavor': 'default', 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-15T00:44:24.263 INFO:teuthology.task.install:Using flavor: default 2024-06-15T00:44:24.268 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-06-15T00:44:24.268 INFO:teuthology.task.install:extra packages: [] 2024-06-15T00:44:24.268 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'tag': None, 'wait_for_package': False} 2024-06-15T00:44:24.269 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:44:24.269 INFO:teuthology.packaging:ref: None 2024-06-15T00:44:24.269 INFO:teuthology.packaging:tag: None 2024-06-15T00:44:24.269 INFO:teuthology.packaging:branch: quincy 2024-06-15T00:44:24.269 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:44:24.269 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-15T00:44:24.271 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': 'quincy', 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc', 'tag': None, 'wait_for_package': False} 2024-06-15T00:44:24.271 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:44:24.272 INFO:teuthology.packaging:ref: None 2024-06-15T00:44:24.272 INFO:teuthology.packaging:tag: None 2024-06-15T00:44:24.272 INFO:teuthology.packaging:branch: quincy 2024-06-15T00:44:24.272 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:44:24.272 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-15T00:44:24.483 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/ 2024-06-15T00:44:24.483 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1008.ge47ff4b8 2024-06-15T00:44:24.501 INFO:teuthology.task.install.rpm:Pulling from https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/ 2024-06-15T00:44:24.501 INFO:teuthology.task.install.rpm:Package version is 17.2.7-1008.ge47ff4b8 2024-06-15T00:44:24.626 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-15T00:44:24.626 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:44:24.626 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-15T00:44:24.652 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-15T00:44:24.652 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:44:24.652 INFO:teuthology.packaging:ref: None 2024-06-15T00:44:24.652 INFO:teuthology.packaging:tag: None 2024-06-15T00:44:24.652 INFO:teuthology.packaging:branch: quincy 2024-06-15T00:44:24.652 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:44:24.653 DEBUG:teuthology.orchestra.run.smithi105:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-15T00:44:24.657 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://2.chacra.ceph.com/r/ceph/quincy/e47ff4b87f747658fc7376cc7065fc56801eb37b/centos/9/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-06-15T00:44:24.657 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:44:24.657 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-06-15T00:44:24.683 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-06-15T00:44:24.683 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:44:24.683 INFO:teuthology.packaging:ref: None 2024-06-15T00:44:24.683 INFO:teuthology.packaging:tag: None 2024-06-15T00:44:24.683 INFO:teuthology.packaging:branch: quincy 2024-06-15T00:44:24.683 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:44:24.683 DEBUG:teuthology.orchestra.run.smithi143:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/quincy/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-06-15T00:44:24.718 DEBUG:teuthology.orchestra.run.smithi105:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-15T00:44:24.749 DEBUG:teuthology.orchestra.run.smithi143:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-06-15T00:44:24.793 DEBUG:teuthology.orchestra.run.smithi105:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-15T00:44:24.823 DEBUG:teuthology.orchestra.run.smithi143:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-06-15T00:44:24.857 INFO:teuthology.orchestra.run.smithi105.stdout:check_obsoletes = 1 2024-06-15T00:44:24.858 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum clean all 2024-06-15T00:44:24.887 INFO:teuthology.orchestra.run.smithi143.stdout:check_obsoletes = 1 2024-06-15T00:44:24.889 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum clean all 2024-06-15T00:44:25.086 INFO:teuthology.orchestra.run.smithi105.stdout:45 files removed 2024-06-15T00:44:25.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-radosgw 2024-06-15T00:44:25.129 INFO:teuthology.orchestra.run.smithi143.stdout:45 files removed 2024-06-15T00:44:25.156 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-radosgw 2024-06-15T00:44:25.838 INFO:teuthology.orchestra.run.smithi143.stdout:ceph packages for x86_64 191 kB/s | 77 kB 00:00 2024-06-15T00:44:25.844 INFO:teuthology.orchestra.run.smithi105.stdout:ceph packages for x86_64 169 kB/s | 77 kB 00:00 2024-06-15T00:44:26.270 INFO:teuthology.orchestra.run.smithi105.stdout:ceph noarch packages 28 kB/s | 11 kB 00:00 2024-06-15T00:44:26.280 INFO:teuthology.orchestra.run.smithi143.stdout:ceph noarch packages 27 kB/s | 11 kB 00:00 2024-06-15T00:44:26.665 INFO:teuthology.orchestra.run.smithi143.stdout:ceph source packages 5.3 kB/s | 1.8 kB 00:00 2024-06-15T00:44:26.695 INFO:teuthology.orchestra.run.smithi105.stdout:ceph source packages 4.7 kB/s | 1.8 kB 00:00 2024-06-15T00:44:27.312 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - BaseOS 14 MB/s | 8.1 MB 00:00 2024-06-15T00:44:28.740 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - BaseOS 4.0 MB/s | 8.1 MB 00:02 2024-06-15T00:44:30.616 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - AppStream 26 MB/s | 20 MB 00:00 2024-06-15T00:44:33.379 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - AppStream 3.9 MB/s | 20 MB 00:04 2024-06-15T00:44:34.767 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - CRB 15 MB/s | 6.3 MB 00:00 2024-06-15T00:44:36.509 INFO:teuthology.orchestra.run.smithi143.stdout:CentOS Stream 9 - Extras packages 39 kB/s | 17 kB 00:00 2024-06-15T00:44:38.272 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - CRB 5.1 MB/s | 6.3 MB 00:01 2024-06-15T00:44:39.931 INFO:teuthology.orchestra.run.smithi105.stdout:CentOS Stream 9 - Extras packages 56 kB/s | 17 kB 00:00 2024-06-15T00:44:41.326 INFO:teuthology.orchestra.run.smithi143.stdout:Extra Packages for Enterprise Linux 4.6 MB/s | 21 MB 00:04 2024-06-15T00:44:44.766 INFO:teuthology.orchestra.run.smithi105.stdout:Extra Packages for Enterprise Linux 4.5 MB/s | 21 MB 00:04 2024-06-15T00:44:46.620 INFO:teuthology.orchestra.run.smithi143.stdout:lab-extras 15 kB/s | 1.7 kB 00:00 2024-06-15T00:44:49.028 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:44:49.031 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:44:49.031 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:44:49.031 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:44:49.031 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:44:49.031 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 14 M 2024-06-15T00:44:49.031 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrading: 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: librados2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.1 M 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 5.0 M 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 22 M 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 638 k 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-15T00:44:49.032 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 457 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 4.9 M 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 45 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 118 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 156 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 320 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 292 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 100 k 2024-06-15T00:44:49.033 INFO:teuthology.orchestra.run.smithi143.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout:Install 22 Packages 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout:Upgrade 2 Packages 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:44:49.034 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 58 M 2024-06-15T00:44:49.035 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:44:50.033 INFO:teuthology.orchestra.run.smithi105.stdout:lab-extras 18 kB/s | 1.7 kB 00:00 2024-06-15T00:44:50.713 INFO:teuthology.orchestra.run.smithi143.stdout:(1/24): ceph-base-17.2.7-1008.ge47ff4b8.el9.x86 4.4 MB/s | 5.0 MB 00:01 2024-06-15T00:44:50.854 INFO:teuthology.orchestra.run.smithi143.stdout:(2/24): ceph-radosgw-17.2.7-1008.ge47ff4b8.el9. 11 MB/s | 14 MB 00:01 2024-06-15T00:44:50.880 INFO:teuthology.orchestra.run.smithi143.stdout:(3/24): ceph-selinux-17.2.7-1008.ge47ff4b8.el9. 151 kB/s | 25 kB 00:00 2024-06-15T00:44:50.914 INFO:teuthology.orchestra.run.smithi143.stdout:(4/24): libcephfs2-17.2.7-1008.ge47ff4b8.el9.x8 11 MB/s | 638 kB 00:00 2024-06-15T00:44:50.980 INFO:teuthology.orchestra.run.smithi143.stdout:(5/24): libradosstriper1-17.2.7-1008.ge47ff4b8. 4.4 MB/s | 457 kB 00:00 2024-06-15T00:44:51.106 INFO:teuthology.orchestra.run.smithi143.stdout:(6/24): python3-ceph-argparse-17.2.7-1008.ge47f 360 kB/s | 45 kB 00:00 2024-06-15T00:44:51.314 INFO:teuthology.orchestra.run.smithi143.stdout:(7/24): librgw2-17.2.7-1008.ge47ff4b8.el9.x86_6 12 MB/s | 4.9 MB 00:00 2024-06-15T00:44:51.390 INFO:teuthology.orchestra.run.smithi143.stdout:(8/24): python3-cephfs-17.2.7-1008.ge47ff4b8.el 2.0 MB/s | 156 kB 00:00 2024-06-15T00:44:51.507 INFO:teuthology.orchestra.run.smithi143.stdout:(9/24): python3-rados-17.2.7-1008.ge47ff4b8.el9 2.7 MB/s | 320 kB 00:00 2024-06-15T00:44:51.590 INFO:teuthology.orchestra.run.smithi143.stdout:(10/24): python3-rbd-17.2.7-1008.ge47ff4b8.el9. 3.4 MB/s | 292 kB 00:00 2024-06-15T00:44:51.649 INFO:teuthology.orchestra.run.smithi143.stdout:(11/24): python3-rgw-17.2.7-1008.ge47ff4b8.el9. 1.7 MB/s | 100 kB 00:00 2024-06-15T00:44:51.699 INFO:teuthology.orchestra.run.smithi143.stdout:(12/24): python3-ceph-common-17.2.7-1008.ge47ff 198 kB/s | 118 kB 00:00 2024-06-15T00:44:51.966 INFO:teuthology.orchestra.run.smithi143.stdout:(13/24): ceph-common-17.2.7-1008.ge47ff4b8.el9. 9.4 MB/s | 22 MB 00:02 2024-06-15T00:44:52.473 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:44:52.487 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:44:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:44:52.487 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:44:52.487 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:44:52.487 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 14 M 2024-06-15T00:44:52.487 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrading: 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: librados2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.1 M 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 5.0 M 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 22 M 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs x86_64 2.9.1-2.el9 epel 309 k 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 638 k 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: liboath x86_64 2.6.7-2.el9 epel 49 k 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj x86_64 1.12.1-1.el9 appstream 160 k 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq x86_64 0.11.0-7.el9 appstream 45 k 2024-06-15T00:44:52.488 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 457 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka x86_64 1.6.1-102.el9 appstream 662 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2 x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 4.9 M 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind x86_64 1.6.2-1.el9 epel 67 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust x86_64 2.12.0-6.el9 appstream 292 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 45 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 118 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 156 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable noarch 0.7.2-27.el9 appstream 42 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 320 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 292 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 100 k 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: thrift x86_64 0.15.0-2.el9 epel 1.6 M 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:44:52.489 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:44:52.490 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:44:52.490 INFO:teuthology.orchestra.run.smithi105.stdout:Install 22 Packages 2024-06-15T00:44:52.490 INFO:teuthology.orchestra.run.smithi105.stdout:Upgrade 2 Packages 2024-06-15T00:44:52.490 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:44:52.490 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 58 M 2024-06-15T00:44:52.490 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:44:52.533 INFO:teuthology.orchestra.run.smithi143.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 54 kB/s | 45 kB 00:00 2024-06-15T00:44:52.600 INFO:teuthology.orchestra.run.smithi143.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 168 kB/s | 160 kB 00:00 2024-06-15T00:44:52.917 INFO:teuthology.orchestra.run.smithi143.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 697 kB/s | 662 kB 00:00 2024-06-15T00:44:52.984 INFO:teuthology.orchestra.run.smithi143.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.5 MB/s | 309 kB 00:00 2024-06-15T00:44:53.001 INFO:teuthology.orchestra.run.smithi143.stdout:(18/24): liboath-2.6.7-2.el9.x86_64.rpm 2.9 MB/s | 49 kB 00:00 2024-06-15T00:44:53.026 INFO:teuthology.orchestra.run.smithi143.stdout:(19/24): libunwind-1.6.2-1.el9.x86_64.rpm 2.6 MB/s | 67 kB 00:00 2024-06-15T00:44:53.085 INFO:teuthology.orchestra.run.smithi143.stdout:(20/24): thrift-0.15.0-2.el9.x86_64.rpm 27 MB/s | 1.6 MB 00:00 2024-06-15T00:44:53.252 INFO:teuthology.orchestra.run.smithi143.stdout:(21/24): python3-prettytable-0.7.2-27.el9.noarc 65 kB/s | 42 kB 00:00 2024-06-15T00:44:53.386 INFO:teuthology.orchestra.run.smithi143.stdout:(22/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 343 kB/s | 292 kB 00:00 2024-06-15T00:44:54.202 INFO:teuthology.orchestra.run.smithi143.stdout:(23/24): librbd1-17.2.7-1008.ge47ff4b8.el9.x86_ 3.1 MB/s | 2.9 MB 00:00 2024-06-15T00:44:54.367 INFO:teuthology.orchestra.run.smithi105.stdout:(1/24): ceph-radosgw-17.2.7-1008.ge47ff4b8.el9. 8.2 MB/s | 14 MB 00:01 2024-06-15T00:44:54.618 INFO:teuthology.orchestra.run.smithi105.stdout:(2/24): ceph-common-17.2.7-1008.ge47ff4b8.el9.x 11 MB/s | 22 MB 00:01 2024-06-15T00:44:54.643 INFO:teuthology.orchestra.run.smithi105.stdout:(3/24): ceph-selinux-17.2.7-1008.ge47ff4b8.el9. 91 kB/s | 25 kB 00:00 2024-06-15T00:44:54.693 INFO:teuthology.orchestra.run.smithi105.stdout:(4/24): libcephfs2-17.2.7-1008.ge47ff4b8.el9.x8 8.3 MB/s | 638 kB 00:00 2024-06-15T00:44:54.744 INFO:teuthology.orchestra.run.smithi105.stdout:(5/24): libradosstriper1-17.2.7-1008.ge47ff4b8. 4.4 MB/s | 457 kB 00:00 2024-06-15T00:44:54.844 INFO:teuthology.orchestra.run.smithi105.stdout:(6/24): ceph-base-17.2.7-1008.ge47ff4b8.el9.x86 2.3 MB/s | 5.0 MB 00:02 2024-06-15T00:44:54.869 INFO:teuthology.orchestra.run.smithi105.stdout:(7/24): python3-ceph-argparse-17.2.7-1008.ge47f 360 kB/s | 45 kB 00:00 2024-06-15T00:44:54.919 INFO:teuthology.orchestra.run.smithi105.stdout:(8/24): python3-ceph-common-17.2.7-1008.ge47ff4 1.5 MB/s | 118 kB 00:00 2024-06-15T00:44:54.944 INFO:teuthology.orchestra.run.smithi105.stdout:(9/24): python3-cephfs-17.2.7-1008.ge47ff4b8.el 2.0 MB/s | 156 kB 00:00 2024-06-15T00:44:55.011 INFO:teuthology.orchestra.run.smithi105.stdout:(10/24): librgw2-17.2.7-1008.ge47ff4b8.el9.x86_ 15 MB/s | 4.9 MB 00:00 2024-06-15T00:44:55.019 INFO:teuthology.orchestra.run.smithi143.stdout:(24/24): librados2-17.2.7-1008.ge47ff4b8.el9.x8 1.6 MB/s | 3.1 MB 00:01 2024-06-15T00:44:55.024 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:44:55.024 INFO:teuthology.orchestra.run.smithi143.stdout:Total 9.6 MB/s | 58 MB 00:05 2024-06-15T00:44:55.037 INFO:teuthology.orchestra.run.smithi105.stdout:(11/24): python3-rbd-17.2.7-1008.ge47ff4b8.el9. 3.1 MB/s | 292 kB 00:00 2024-06-15T00:44:55.062 INFO:teuthology.orchestra.run.smithi105.stdout:(12/24): python3-rados-17.2.7-1008.ge47ff4b8.el 2.2 MB/s | 320 kB 00:00 2024-06-15T00:44:55.086 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:44:55.087 INFO:teuthology.orchestra.run.smithi105.stdout:(13/24): python3-rgw-17.2.7-1008.ge47ff4b8.el9. 1.3 MB/s | 100 kB 00:00 2024-06-15T00:44:55.120 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:44:55.120 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:44:55.237 INFO:teuthology.orchestra.run.smithi105.stdout:(14/24): librabbitmq-0.11.0-7.el9.x86_64.rpm 258 kB/s | 45 kB 00:00 2024-06-15T00:44:55.263 INFO:teuthology.orchestra.run.smithi105.stdout:(15/24): libpmemobj-1.12.1-1.el9.x86_64.rpm 709 kB/s | 160 kB 00:00 2024-06-15T00:44:55.355 INFO:teuthology.orchestra.run.smithi105.stdout:(16/24): librdkafka-1.6.1-102.el9.x86_64.rpm 2.4 MB/s | 662 kB 00:00 2024-06-15T00:44:55.422 INFO:teuthology.orchestra.run.smithi105.stdout:(17/24): gperftools-libs-2.9.1-2.el9.x86_64.rpm 4.5 MB/s | 309 kB 00:00 2024-06-15T00:44:55.447 INFO:teuthology.orchestra.run.smithi105.stdout:(18/24): python3-prettytable-0.7.2-27.el9.noarc 230 kB/s | 42 kB 00:00 2024-06-15T00:44:55.472 INFO:teuthology.orchestra.run.smithi105.stdout:(19/24): liboath-2.6.7-2.el9.x86_64.rpm 968 kB/s | 49 kB 00:00 2024-06-15T00:44:55.531 INFO:teuthology.orchestra.run.smithi105.stdout:(20/24): lttng-ust-2.12.0-6.el9.x86_64.rpm 998 kB/s | 292 kB 00:00 2024-06-15T00:44:55.564 INFO:teuthology.orchestra.run.smithi105.stdout:(21/24): libunwind-1.6.2-1.el9.x86_64.rpm 575 kB/s | 67 kB 00:00 2024-06-15T00:44:55.606 INFO:teuthology.orchestra.run.smithi105.stdout:(22/24): thrift-0.15.0-2.el9.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2024-06-15T00:44:55.659 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:44:55.659 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:44:55.782 INFO:teuthology.orchestra.run.smithi105.stdout:(23/24): librbd1-17.2.7-1008.ge47ff4b8.el9.x86_ 14 MB/s | 2.9 MB 00:00 2024-06-15T00:44:56.032 INFO:teuthology.orchestra.run.smithi105.stdout:(24/24): librados2-17.2.7-1008.ge47ff4b8.el9.x8 6.2 MB/s | 3.1 MB 00:00 2024-06-15T00:44:56.037 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:44:56.037 INFO:teuthology.orchestra.run.smithi105.stdout:Total 16 MB/s | 58 MB 00:03 2024-06-15T00:44:56.102 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:44:56.135 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:44:56.135 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:44:56.660 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:44:56.660 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:44:56.867 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:44:57.250 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-15T00:44:57.322 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-15T00:44:57.413 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-15T00:44:57.493 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-15T00:44:57.515 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-15T00:44:57.606 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-15T00:44:57.658 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-15T00:44:57.687 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-15T00:44:57.867 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:44:58.119 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-15T00:44:58.144 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-15T00:44:58.206 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-15T00:44:58.235 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-15T00:44:58.264 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : lttng-ust-2.12.0-6.el9.x86_64 1/26 2024-06-15T00:44:58.278 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 10/26 2024-06-15T00:44:58.309 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-15T00:44:58.316 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-15T00:44:58.377 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-15T00:44:58.385 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-15T00:44:58.405 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-15T00:44:58.462 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-15T00:44:58.464 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-15T00:44:58.502 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-15T00:44:58.503 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-15T00:44:58.539 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-15T00:44:58.571 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-15T00:44:58.582 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-15T00:44:58.628 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : thrift-0.15.0-2.el9.x86_64 5/26 2024-06-15T00:44:58.657 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librdkafka-1.6.1-102.el9.x86_64 6/26 2024-06-15T00:44:58.866 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-15T00:44:58.910 INFO:teuthology.orchestra.run.smithi143.stdout: Upgrading : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-15T00:44:59.000 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-15T00:44:59.073 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 19/26 2024-06-15T00:44:59.091 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librabbitmq-0.11.0-7.el9.x86_64 7/26 2024-06-15T00:44:59.104 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 20/26 2024-06-15T00:44:59.122 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-15T00:44:59.185 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 8/26 2024-06-15T00:44:59.213 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : liboath-2.6.7-2.el9.x86_64 9/26 2024-06-15T00:44:59.256 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 10/26 2024-06-15T00:44:59.293 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-15T00:44:59.356 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-15T00:44:59.383 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-15T00:44:59.452 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 13/26 2024-06-15T00:44:59.494 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libunwind-1.6.2-1.el9.x86_64 14/26 2024-06-15T00:44:59.533 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : gperftools-libs-2.9.1-2.el9.x86_64 15/26 2024-06-15T00:44:59.565 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-prettytable-0.7.2-27.el9.noarch 16/26 2024-06-15T00:44:59.862 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libpmemobj-1.12.1-1.el9.x86_64 17/26 2024-06-15T00:44:59.903 INFO:teuthology.orchestra.run.smithi105.stdout: Upgrading : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-15T00:44:59.977 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 18/26 2024-06-15T00:45:00.051 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 19/26 2024-06-15T00:45:00.081 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 20/26 2024-06-15T00:45:01.589 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-15T00:45:01.627 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-15T00:45:02.168 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-15T00:45:02.214 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-15T00:45:02.308 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-15T00:45:02.309 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-15T00:45:02.309 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-15T00:45:02.309 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:02.341 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-15T00:45:02.423 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-15T00:45:02.464 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-15T00:45:02.995 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 21/26 2024-06-15T00:45:03.034 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-15T00:45:03.121 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 22/26 2024-06-15T00:45:03.121 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /usr/lib/systemd/system/ceph.target. 2024-06-15T00:45:03.121 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service → /usr/lib/systemd/system/ceph-crash.service. 2024-06-15T00:45:03.121 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:03.144 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-15T00:45:11.725 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-15T00:45:11.725 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /sys 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /proc 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /mnt 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /var/tmp 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /home 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /root 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout:skipping the directory /tmp 2024-06-15T00:45:11.726 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:12.561 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-15T00:45:12.561 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /sys 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /proc 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /mnt 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /var/tmp 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /home 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /root 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout:skipping the directory /tmp 2024-06-15T00:45:12.562 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:13.014 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-15T00:45:13.083 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-15T00:45:13.084 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:13.084 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-15T00:45:13.084 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-15T00:45:13.084 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-15T00:45:13.084 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:13.085 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-15T00:45:13.167 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-15T00:45:13.167 INFO:teuthology.orchestra.run.smithi143.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-15T00:45:13.863 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-15T00:45:13.942 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 24/26 2024-06-15T00:45:13.942 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:13.942 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-radosgw@*.service" escaped as "ceph-radosgw@\x2a.service". 2024-06-15T00:45:13.942 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-15T00:45:13.942 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target → /usr/lib/systemd/system/ceph-radosgw.target. 2024-06-15T00:45:13.942 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:13.944 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-15T00:45:14.016 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librbd1-2:16.2.4-5.el9.x86_64 25/26 2024-06-15T00:45:14.016 INFO:teuthology.orchestra.run.smithi105.stdout: Cleanup : librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 6/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 7/26 2024-06-15T00:45:14.474 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 8/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 9/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 10/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 13/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-15T00:45:14.475 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-15T00:45:14.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-15T00:45:14.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-15T00:45:14.476 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 25/26 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout:Upgraded: 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.153 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.154 INFO:teuthology.orchestra.run.smithi143.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:15.155 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:15.303 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados2-2:16.2.4-5.el9.x86_64 26/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_6 6/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 7/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9. 8/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x8 9/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 10/26 2024-06-15T00:45:15.304 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 11/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 12/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 13/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libpmemobj-1.12.1-1.el9.x86_64 14/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librabbitmq-0.11.0-7.el9.x86_64 15/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librdkafka-1.6.1-102.el9.x86_64 16/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : lttng-ust-2.12.0-6.el9.x86_64 17/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-prettytable-0.7.2-27.el9.noarch 18/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : gperftools-libs-2.9.1-2.el9.x86_64 19/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : liboath-2.6.7-2.el9.x86_64 20/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libunwind-1.6.2-1.el9.x86_64 21/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : thrift-0.15.0-2.el9.x86_64 22/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 23/26 2024-06-15T00:45:15.305 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados2-2:16.2.4-5.el9.x86_64 24/26 2024-06-15T00:45:15.306 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 25/26 2024-06-15T00:45:15.543 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-test 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librbd1-2:16.2.4-5.el9.x86_64 26/26 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout:Upgraded: 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout: librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout: librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:45:15.944 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-radosgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-selinux-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: gperftools-libs-2.9.1-2.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: liboath-2.6.7-2.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: libpmemobj-1.12.1-1.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: librabbitmq-0.11.0-7.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: libradosstriper1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: librdkafka-1.6.1-102.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: librgw2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: libunwind-1.6.2-1.el9.x86_64 2024-06-15T00:45:15.945 INFO:teuthology.orchestra.run.smithi105.stdout: lttng-ust-2.12.0-6.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-argparse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ceph-common-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-prettytable-0.7.2-27.el9.noarch 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: thrift-0.15.0-2.el9.x86_64 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:15.946 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:16.123 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:30 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:16.249 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:16.249 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:16.249 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 48 M 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 163 k 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:16.250 INFO:teuthology.orchestra.run.smithi143.stdout:Install 4 Packages 2024-06-15T00:45:16.251 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:16.251 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 48 M 2024-06-15T00:45:16.251 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 192 M 2024-06-15T00:45:16.251 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:45:16.302 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-test 2024-06-15T00:45:16.507 INFO:teuthology.orchestra.run.smithi143.stdout:(1/4): libcephsqlite-17.2.7-1008.ge47ff4b8.el9. 750 kB/s | 163 kB 00:00 2024-06-15T00:45:16.849 INFO:teuthology.orchestra.run.smithi143.stdout:(2/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 186 kB/s | 64 kB 00:00 2024-06-15T00:45:16.874 INFO:teuthology.orchestra.run.smithi143.stdout:(3/4): socat-1.7.4.1-6.el9.x86_64.rpm 520 kB/s | 304 kB 00:00 2024-06-15T00:45:16.885 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:26 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:17.010 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 48 M 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 163 k 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout: socat x86_64 1.7.4.1-6.el9 appstream 304 k 2024-06-15T00:45:17.011 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet x86_64 1.6.1-20.el9 appstream 64 k 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout:Install 4 Packages 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 48 M 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 192 M 2024-06-15T00:45:17.012 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:45:17.495 INFO:teuthology.orchestra.run.smithi105.stdout:(1/4): libcephsqlite-17.2.7-1008.ge47ff4b8.el9. 425 kB/s | 163 kB 00:00 2024-06-15T00:45:17.753 INFO:teuthology.orchestra.run.smithi105.stdout:(2/4): socat-1.7.4.1-6.el9.x86_64.rpm 474 kB/s | 304 kB 00:00 2024-06-15T00:45:17.787 INFO:teuthology.orchestra.run.smithi105.stdout:(3/4): xmlstarlet-1.6.1-20.el9.x86_64.rpm 218 kB/s | 64 kB 00:00 2024-06-15T00:45:20.274 INFO:teuthology.orchestra.run.smithi143.stdout:(4/4): ceph-test-17.2.7-1008.ge47ff4b8.el9.x86_ 12 MB/s | 48 MB 00:03 2024-06-15T00:45:20.276 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:20.277 INFO:teuthology.orchestra.run.smithi143.stdout:Total 12 MB/s | 48 MB 00:04 2024-06-15T00:45:20.303 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:45:20.322 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:45:20.322 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:45:20.688 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:45:20.688 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:45:20.804 INFO:teuthology.orchestra.run.smithi105.stdout:(4/4): ceph-test-17.2.7-1008.ge47ff4b8.el9.x86_ 13 MB/s | 48 MB 00:03 2024-06-15T00:45:20.807 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:20.807 INFO:teuthology.orchestra.run.smithi105.stdout:Total 13 MB/s | 48 MB 00:03 2024-06-15T00:45:20.833 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:45:20.852 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:45:20.852 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:45:21.215 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:45:21.217 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:45:21.218 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:45:21.293 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-15T00:45:21.343 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-15T00:45:21.368 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-15T00:45:21.738 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:45:21.813 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : xmlstarlet-1.6.1-20.el9.x86_64 1/4 2024-06-15T00:45:21.862 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : socat-1.7.4.1-6.el9.x86_64 2/4 2024-06-15T00:45:21.889 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-15T00:45:25.883 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-15T00:45:25.975 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-15T00:45:26.374 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/4 2024-06-15T00:45:26.447 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-15T00:45:28.145 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-15T00:45:28.145 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/4 2024-06-15T00:45:28.145 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/4 2024-06-15T00:45:28.145 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-15T00:45:28.412 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:28.413 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:28.702 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph 2024-06-15T00:45:28.709 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/4 2024-06-15T00:45:28.709 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/4 2024-06-15T00:45:28.709 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/4 2024-06-15T00:45:28.709 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : socat-1.7.4.1-6.el9.x86_64 3/4 2024-06-15T00:45:28.989 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : xmlstarlet-1.6.1-20.el9.x86_64 4/4 2024-06-15T00:45:28.989 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:28.989 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:45:28.989 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-test-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:28.989 INFO:teuthology.orchestra.run.smithi105.stdout: libcephsqlite-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:28.989 INFO:teuthology.orchestra.run.smithi105.stdout: socat-1.7.4.1-6.el9.x86_64 2024-06-15T00:45:28.990 INFO:teuthology.orchestra.run.smithi105.stdout: xmlstarlet-1.6.1-20.el9.x86_64 2024-06-15T00:45:28.990 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:28.990 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:29.293 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:43 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:29.297 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph 2024-06-15T00:45:29.428 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:29.430 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:29.430 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: ceph x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 6.5 k 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.1 M 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 1.4 M 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 240 k 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.9 M 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 15 M 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-15T00:45:29.431 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-15T00:45:29.432 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-15T00:45:29.433 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout:Install 33 Packages 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 27 M 2024-06-15T00:45:29.434 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 96 M 2024-06-15T00:45:29.435 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:45:29.870 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:39 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:30.001 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: ceph x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 6.5 k 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.1 M 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 1.4 M 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 240 k 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 3.9 M 2024-06-15T00:45:30.004 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 15 M 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt x86_64 3.2.2-1.el9 epel 43 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi x86_64 1.14.5-5.el9 baseos 253 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot noarch 8.6.0-4.el9 epel 172 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy noarch 18.6.1-2.el9 epel 358 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography x86_64 36.0.1-4.el9 baseos 1.2 M 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco noarch 8.2.1-3.el9 epel 11 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes noarch 3.2.1-5.el9 epel 18 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: noarch 3.0.0-8.el9 epel 23 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools noarch 3.5.0-2.el9 epel 19 k 2024-06-15T00:45:30.005 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text noarch 3.2.0-6.el9 epel 20 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils noarch 0.3.5-21.el9 epel 46 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako noarch 1.1.4-6.el9 appstream 172 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe x86_64 1.1.1-12.el9 appstream 35 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools noarch 8.12.0-2.el9 epel 79 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan noarch 1.4.2-3.el9 epel 272 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply noarch 3.11-14.el9 baseos 106 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend noarch 3.1.0-2.el9 epel 16 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL noarch 21.0.0-1.el9 epel 90 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser noarch 2.20-6.el9 baseos 135 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz noarch 2021.1-5.el9 appstream 51 k 2024-06-15T00:45:30.006 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests noarch 2.25.1-8.el9 baseos 125 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora noarch 5.0.0-2.el9 epel 36 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml noarch 0.10.2-6.el9 appstream 42 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3 noarch 1.26.5-5.el9 baseos 215 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob noarch 1.8.7-6.el9 epel 230 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug noarch 2.0.3-3.el9.1 epel 427 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile noarch 2.0-10.el9 epel 20 k 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout:Install 33 Packages 2024-06-15T00:45:30.007 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:30.008 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 27 M 2024-06-15T00:45:30.008 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 96 M 2024-06-15T00:45:30.008 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:45:30.286 INFO:teuthology.orchestra.run.smithi143.stdout:(1/33): ceph-17.2.7-1008.ge47ff4b8.el9.x86_64.r 25 kB/s | 6.5 kB 00:00 2024-06-15T00:45:30.411 INFO:teuthology.orchestra.run.smithi143.stdout:(2/33): ceph-mgr-17.2.7-1008.ge47ff4b8.el9.x86_ 3.7 MB/s | 1.4 MB 00:00 2024-06-15T00:45:30.688 INFO:teuthology.orchestra.run.smithi105.stdout:(1/33): ceph-17.2.7-1008.ge47ff4b8.el9.x86_64.r 21 kB/s | 6.5 kB 00:00 2024-06-15T00:45:30.911 INFO:teuthology.orchestra.run.smithi143.stdout:(3/33): ceph-mds-17.2.7-1008.ge47ff4b8.el9.x86_ 2.3 MB/s | 2.1 MB 00:00 2024-06-15T00:45:31.037 INFO:teuthology.orchestra.run.smithi143.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1008.ge47f 1.9 MB/s | 240 kB 00:00 2024-06-15T00:45:31.255 INFO:teuthology.orchestra.run.smithi105.stdout:(2/33): ceph-mgr-17.2.7-1008.ge47ff4b8.el9.x86_ 1.6 MB/s | 1.4 MB 00:00 2024-06-15T00:45:31.262 INFO:teuthology.orchestra.run.smithi143.stdout:(5/33): ceph-osd-17.2.7-1008.ge47ff4b8.el9.x86_ 18 MB/s | 15 MB 00:00 2024-06-15T00:45:31.464 INFO:teuthology.orchestra.run.smithi105.stdout:(3/33): ceph-mds-17.2.7-1008.ge47ff4b8.el9.x86_ 1.9 MB/s | 2.1 MB 00:01 2024-06-15T00:45:31.531 INFO:teuthology.orchestra.run.smithi105.stdout:(4/33): ceph-mgr-modules-core-17.2.7-1008.ge47f 3.5 MB/s | 240 kB 00:00 2024-06-15T00:45:31.687 INFO:teuthology.orchestra.run.smithi143.stdout:(6/33): ceph-mon-17.2.7-1008.ge47ff4b8.el9.x86_ 2.8 MB/s | 3.9 MB 00:01 2024-06-15T00:45:31.806 INFO:teuthology.orchestra.run.smithi105.stdout:(5/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 921 kB/s | 253 kB 00:00 2024-06-15T00:45:32.015 INFO:teuthology.orchestra.run.smithi105.stdout:(6/33): python3-cryptography-36.0.1-4.el9.x86_6 5.9 MB/s | 1.2 MB 00:00 2024-06-15T00:45:32.020 INFO:teuthology.orchestra.run.smithi143.stdout:(7/33): python3-cffi-1.14.5-5.el9.x86_64.rpm 257 kB/s | 253 kB 00:00 2024-06-15T00:45:32.107 INFO:teuthology.orchestra.run.smithi105.stdout:(7/33): python3-ply-3.11-14.el9.noarch.rpm 1.1 MB/s | 106 kB 00:00 2024-06-15T00:45:32.191 INFO:teuthology.orchestra.run.smithi105.stdout:(8/33): ceph-mon-17.2.7-1008.ge47ff4b8.el9.x86_ 2.6 MB/s | 3.9 MB 00:01 2024-06-15T00:45:32.304 INFO:teuthology.orchestra.run.smithi143.stdout:(8/33): python3-cryptography-36.0.1-4.el9.x86_6 1.2 MB/s | 1.2 MB 00:01 2024-06-15T00:45:32.341 INFO:teuthology.orchestra.run.smithi105.stdout:(9/33): ceph-osd-17.2.7-1008.ge47ff4b8.el9.x86_ 14 MB/s | 15 MB 00:01 2024-06-15T00:45:32.366 INFO:teuthology.orchestra.run.smithi105.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 522 kB/s | 135 kB 00:00 2024-06-15T00:45:32.446 INFO:teuthology.orchestra.run.smithi143.stdout:(9/33): python3-ply-3.11-14.el9.noarch.rpm 140 kB/s | 106 kB 00:00 2024-06-15T00:45:32.475 INFO:teuthology.orchestra.run.smithi105.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-06-15T00:45:32.517 INFO:teuthology.orchestra.run.smithi105.stdout:(12/33): python3-requests-2.25.1-8.el9.noarch.r 383 kB/s | 125 kB 00:00 2024-06-15T00:45:32.542 INFO:teuthology.orchestra.run.smithi105.stdout:(13/33): python3-urllib3-1.26.5-5.el9.noarch.rp 1.0 MB/s | 215 kB 00:00 2024-06-15T00:45:32.567 INFO:teuthology.orchestra.run.smithi105.stdout:(14/33): python3-markupsafe-1.1.1-12.el9.x86_64 377 kB/s | 35 kB 00:00 2024-06-15T00:45:32.592 INFO:teuthology.orchestra.run.smithi105.stdout:(15/33): python3-toml-0.10.2-6.el9.noarch.rpm 832 kB/s | 42 kB 00:00 2024-06-15T00:45:32.618 INFO:teuthology.orchestra.run.smithi105.stdout:(16/33): python3-pytz-2021.1-5.el9.noarch.rpm 506 kB/s | 51 kB 00:00 2024-06-15T00:45:32.718 INFO:teuthology.orchestra.run.smithi105.stdout:(17/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 288 kB/s | 43 kB 00:00 2024-06-15T00:45:32.743 INFO:teuthology.orchestra.run.smithi105.stdout:(18/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 470 kB/s | 11 kB 00:00 2024-06-15T00:45:32.769 INFO:teuthology.orchestra.run.smithi105.stdout:(19/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 977 kB/s | 172 kB 00:00 2024-06-15T00:45:32.788 INFO:teuthology.orchestra.run.smithi143.stdout:(10/33): python3-pycparser-2.20-6.el9.noarch.rp 176 kB/s | 135 kB 00:00 2024-06-15T00:45:32.794 INFO:teuthology.orchestra.run.smithi105.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 354 kB/s | 18 kB 00:00 2024-06-15T00:45:32.819 INFO:teuthology.orchestra.run.smithi105.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 461 kB/s | 23 kB 00:00 2024-06-15T00:45:32.844 INFO:teuthology.orchestra.run.smithi105.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 387 kB/s | 19 kB 00:00 2024-06-15T00:45:32.861 INFO:teuthology.orchestra.run.smithi105.stdout:(23/33): python3-jaraco-text-3.2.0-6.el9.noarch 469 kB/s | 20 kB 00:00 2024-06-15T00:45:32.878 INFO:teuthology.orchestra.run.smithi105.stdout:(24/33): python3-logutils-0.3.5-21.el9.noarch.r 1.3 MB/s | 46 kB 00:00 2024-06-15T00:45:32.903 INFO:teuthology.orchestra.run.smithi105.stdout:(25/33): python3-more-itertools-8.12.0-2.el9.no 1.8 MB/s | 79 kB 00:00 2024-06-15T00:45:32.905 INFO:teuthology.orchestra.run.smithi143.stdout:(11/33): python3-mako-1.1.4-6.el9.noarch.rpm 1.5 MB/s | 172 kB 00:00 2024-06-15T00:45:32.929 INFO:teuthology.orchestra.run.smithi105.stdout:(26/33): python3-cherrypy-18.6.1-2.el9.noarch.r 1.3 MB/s | 358 kB 00:00 2024-06-15T00:45:32.930 INFO:teuthology.orchestra.run.smithi143.stdout:(12/33): python3-markupsafe-1.1.1-12.el9.x86_64 1.4 MB/s | 35 kB 00:00 2024-06-15T00:45:32.946 INFO:teuthology.orchestra.run.smithi105.stdout:(27/33): python3-portend-3.1.0-2.el9.noarch.rpm 390 kB/s | 16 kB 00:00 2024-06-15T00:45:32.966 INFO:teuthology.orchestra.run.smithi143.stdout:(13/33): python3-pytz-2021.1-5.el9.noarch.rpm 1.4 MB/s | 51 kB 00:00 2024-06-15T00:45:32.971 INFO:teuthology.orchestra.run.smithi105.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 2.1 MB/s | 90 kB 00:00 2024-06-15T00:45:32.996 INFO:teuthology.orchestra.run.smithi105.stdout:(29/33): python3-tempora-5.0.0-2.el9.noarch.rpm 714 kB/s | 36 kB 00:00 2024-06-15T00:45:33.000 INFO:teuthology.orchestra.run.smithi143.stdout:(14/33): python3-toml-0.10.2-6.el9.noarch.rpm 1.2 MB/s | 42 kB 00:00 2024-06-15T00:45:33.023 INFO:teuthology.orchestra.run.smithi105.stdout:(30/33): python3-pecan-1.4.2-3.el9.noarch.rpm 1.8 MB/s | 272 kB 00:00 2024-06-15T00:45:33.049 INFO:teuthology.orchestra.run.smithi105.stdout:(31/33): python3-webob-1.8.7-6.el9.noarch.rpm 2.9 MB/s | 230 kB 00:00 2024-06-15T00:45:33.050 INFO:teuthology.orchestra.run.smithi143.stdout:(15/33): python3-requests-2.25.1-8.el9.noarch.r 167 kB/s | 125 kB 00:00 2024-06-15T00:45:33.074 INFO:teuthology.orchestra.run.smithi105.stdout:(32/33): python3-zc-lockfile-2.0-10.el9.noarch. 398 kB/s | 20 kB 00:00 2024-06-15T00:45:33.099 INFO:teuthology.orchestra.run.smithi105.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 4.1 MB/s | 427 kB 00:00 2024-06-15T00:45:33.100 INFO:teuthology.orchestra.run.smithi143.stdout:(16/33): python3-bcrypt-3.2.2-1.el9.x86_64.rpm 434 kB/s | 43 kB 00:00 2024-06-15T00:45:33.103 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:33.103 INFO:teuthology.orchestra.run.smithi105.stdout:Total 8.8 MB/s | 27 MB 00:03 2024-06-15T00:45:33.134 INFO:teuthology.orchestra.run.smithi143.stdout:(17/33): python3-cherrypy-18.6.1-2.el9.noarch.r 11 MB/s | 358 kB 00:00 2024-06-15T00:45:33.159 INFO:teuthology.orchestra.run.smithi143.stdout:(18/33): python3-cheroot-8.6.0-4.el9.noarch.rpm 1.6 MB/s | 172 kB 00:00 2024-06-15T00:45:33.184 INFO:teuthology.orchestra.run.smithi143.stdout:(19/33): python3-jaraco-8.2.1-3.el9.noarch.rpm 213 kB/s | 11 kB 00:00 2024-06-15T00:45:33.209 INFO:teuthology.orchestra.run.smithi143.stdout:(20/33): python3-jaraco-classes-3.2.1-5.el9.noa 710 kB/s | 18 kB 00:00 2024-06-15T00:45:33.216 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:45:33.235 INFO:teuthology.orchestra.run.smithi143.stdout:(21/33): python3-jaraco-collections-3.0.0-8.el9 462 kB/s | 23 kB 00:00 2024-06-15T00:45:33.255 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:45:33.255 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:45:33.260 INFO:teuthology.orchestra.run.smithi143.stdout:(22/33): python3-jaraco-functools-3.5.0-2.el9.n 386 kB/s | 19 kB 00:00 2024-06-15T00:45:33.285 INFO:teuthology.orchestra.run.smithi143.stdout:(23/33): python3-urllib3-1.26.5-5.el9.noarch.rp 256 kB/s | 215 kB 00:00 2024-06-15T00:45:33.311 INFO:teuthology.orchestra.run.smithi143.stdout:(24/33): python3-jaraco-text-3.2.0-6.el9.noarch 260 kB/s | 20 kB 00:00 2024-06-15T00:45:33.361 INFO:teuthology.orchestra.run.smithi143.stdout:(25/33): python3-logutils-0.3.5-21.el9.noarch.r 464 kB/s | 46 kB 00:00 2024-06-15T00:45:33.386 INFO:teuthology.orchestra.run.smithi143.stdout:(26/33): python3-portend-3.1.0-2.el9.noarch.rpm 719 kB/s | 16 kB 00:00 2024-06-15T00:45:33.411 INFO:teuthology.orchestra.run.smithi143.stdout:(27/33): python3-more-itertools-8.12.0-2.el9.no 627 kB/s | 79 kB 00:00 2024-06-15T00:45:33.436 INFO:teuthology.orchestra.run.smithi143.stdout:(28/33): python3-pyOpenSSL-21.0.0-1.el9.noarch. 1.8 MB/s | 90 kB 00:00 2024-06-15T00:45:33.462 INFO:teuthology.orchestra.run.smithi143.stdout:(29/33): python3-pecan-1.4.2-3.el9.noarch.rpm 2.1 MB/s | 272 kB 00:00 2024-06-15T00:45:33.487 INFO:teuthology.orchestra.run.smithi143.stdout:(30/33): python3-tempora-5.0.0-2.el9.noarch.rpm 474 kB/s | 36 kB 00:00 2024-06-15T00:45:33.529 INFO:teuthology.orchestra.run.smithi143.stdout:(31/33): python3-zc-lockfile-2.0-10.el9.noarch. 479 kB/s | 20 kB 00:00 2024-06-15T00:45:33.554 INFO:teuthology.orchestra.run.smithi143.stdout:(32/33): python3-webob-1.8.7-6.el9.noarch.rpm 1.9 MB/s | 230 kB 00:00 2024-06-15T00:45:33.579 INFO:teuthology.orchestra.run.smithi143.stdout:(33/33): python3-werkzeug-2.0.3-3.el9.1.noarch. 3.6 MB/s | 427 kB 00:00 2024-06-15T00:45:33.584 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:33.585 INFO:teuthology.orchestra.run.smithi143.stdout:Total 6.5 MB/s | 27 MB 00:04 2024-06-15T00:45:33.625 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:45:33.626 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:45:33.699 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:45:33.736 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:45:33.736 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:45:34.122 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:45:34.123 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:45:34.755 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:45:34.827 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-15T00:45:34.859 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-15T00:45:34.901 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-15T00:45:34.991 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-15T00:45:35.069 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-15T00:45:35.111 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-15T00:45:35.154 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-15T00:45:35.184 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-15T00:45:35.219 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-15T00:45:35.252 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-15T00:45:35.281 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:45:35.292 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-15T00:45:35.336 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-15T00:45:35.349 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-more-itertools-8.12.0-2.el9.noarch 1/33 2024-06-15T00:45:35.379 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-15T00:45:35.389 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-8.2.1-3.el9.noarch 2/33 2024-06-15T00:45:35.417 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-15T00:45:35.422 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-classes-3.2.1-5.el9.noarch 3/33 2024-06-15T00:45:35.497 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-15T00:45:35.515 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-zc-lockfile-2.0-10.el9.noarch 4/33 2024-06-15T00:45:35.593 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-werkzeug-2.0.3-3.el9.1.noarch 5/33 2024-06-15T00:45:35.633 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-webob-1.8.7-6.el9.noarch 6/33 2024-06-15T00:45:35.634 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-15T00:45:35.676 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-logutils-0.3.5-21.el9.noarch 7/33 2024-06-15T00:45:35.706 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-toml-0.10.2-6.el9.noarch 8/33 2024-06-15T00:45:35.715 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-15T00:45:35.741 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-functools-3.5.0-2.el9.noarch 9/33 2024-06-15T00:45:35.769 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-15T00:45:35.774 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-text-3.2.0-6.el9.noarch 10/33 2024-06-15T00:45:35.808 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-15T00:45:35.815 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jaraco-collections-3.0.0-8.el9.noarch 11/33 2024-06-15T00:45:35.850 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pytz-2021.1-5.el9.noarch 12/33 2024-06-15T00:45:35.857 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-15T00:45:35.884 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-tempora-5.0.0-2.el9.noarch 13/33 2024-06-15T00:45:35.920 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-15T00:45:35.922 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-portend-3.1.0-2.el9.noarch 14/33 2024-06-15T00:45:35.995 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-markupsafe-1.1.1-12.el9.x86_64 15/33 2024-06-15T00:45:36.109 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-15T00:45:36.143 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-mako-1.1.4-6.el9.noarch 16/33 2024-06-15T00:45:36.163 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-15T00:45:36.221 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pecan-1.4.2-3.el9.noarch 17/33 2024-06-15T00:45:36.231 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-15T00:45:36.275 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-urllib3-1.26.5-5.el9.noarch 18/33 2024-06-15T00:45:36.313 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-2.25.1-8.el9.noarch 19/33 2024-06-15T00:45:36.319 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-15T00:45:36.364 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-15T00:45:36.370 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-ply-3.11-14.el9.noarch 20/33 2024-06-15T00:45:36.434 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pycparser-2.20-6.el9.noarch 21/33 2024-06-15T00:45:36.483 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-15T00:45:36.629 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cffi-1.14.5-5.el9.x86_64 22/33 2024-06-15T00:45:36.632 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 28/33 2024-06-15T00:45:36.664 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-15T00:45:36.675 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cryptography-36.0.1-4.el9.x86_64 23/33 2024-06-15T00:45:36.690 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-15T00:45:36.690 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:36.690 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-15T00:45:36.690 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-15T00:45:36.690 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-15T00:45:36.691 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:36.753 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyOpenSSL-21.0.0-1.el9.noarch 24/33 2024-06-15T00:45:36.850 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cheroot-8.6.0-4.el9.noarch 25/33 2024-06-15T00:45:36.896 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cherrypy-18.6.1-2.el9.noarch 26/33 2024-06-15T00:45:37.015 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-bcrypt-3.2.2-1.el9.x86_64 27/33 2024-06-15T00:45:37.163 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 28/33 2024-06-15T00:45:37.193 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-15T00:45:37.219 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 29/33 2024-06-15T00:45:37.219 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:37.219 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mgr@*.service" escaped as "ceph-mgr@\x2a.service". 2024-06-15T00:45:37.219 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-15T00:45:37.219 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target → /usr/lib/systemd/system/ceph-mgr.target. 2024-06-15T00:45:37.219 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:37.998 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-15T00:45:38.025 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-15T00:45:38.025 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:38.026 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-15T00:45:38.026 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-15T00:45:38.026 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-15T00:45:38.026 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:38.408 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-15T00:45:38.431 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-15T00:45:38.431 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:38.431 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-15T00:45:38.431 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-15T00:45:38.432 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-15T00:45:38.432 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:38.535 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-15T00:45:38.563 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 30/33 2024-06-15T00:45:38.563 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:38.563 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-osd@*.service" escaped as "ceph-osd@\x2a.service". 2024-06-15T00:45:38.563 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-15T00:45:38.563 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target → /usr/lib/systemd/system/ceph-osd.target. 2024-06-15T00:45:38.563 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:38.633 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-15T00:45:38.657 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-15T00:45:38.657 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:38.657 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-15T00:45:38.657 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-15T00:45:38.657 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-15T00:45:38.657 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:38.721 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-15T00:45:38.945 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-15T00:45:38.969 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 31/33 2024-06-15T00:45:38.969 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:38.969 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mon@*.service" escaped as "ceph-mon@\x2a.service". 2024-06-15T00:45:38.969 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-15T00:45:38.969 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target → /usr/lib/systemd/system/ceph-mon.target. 2024-06-15T00:45:38.969 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:39.170 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-15T00:45:39.194 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 32/33 2024-06-15T00:45:39.194 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:39.194 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-mds@*.service" escaped as "ceph-mds@\x2a.service". 2024-06-15T00:45:39.194 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-15T00:45:39.194 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target → /usr/lib/systemd/system/ceph-mds.target. 2024-06-15T00:45:39.194 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:39.259 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-15T00:45:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-15T00:45:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/33 2024-06-15T00:45:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/33 2024-06-15T00:45:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/33 2024-06-15T00:45:40.453 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 6/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-15T00:45:40.454 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-15T00:45:40.455 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-15T00:45:40.456 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-15T00:45:40.456 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-15T00:45:40.456 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-15T00:45:40.456 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-15T00:45:40.456 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-15T00:45:40.809 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 33/33 2024-06-15T00:45:40.809 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/33 2024-06-15T00:45:40.809 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 3/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 4/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 5/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9. 6/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cffi-1.14.5-5.el9.x86_64 7/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cryptography-36.0.1-4.el9.x86_64 8/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-ply-3.11-14.el9.noarch 9/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pycparser-2.20-6.el9.noarch 10/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-2.25.1-8.el9.noarch 11/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-urllib3-1.26.5-5.el9.noarch 12/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-mako-1.1.4-6.el9.noarch 13/33 2024-06-15T00:45:40.810 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-markupsafe-1.1.1-12.el9.x86_64 14/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pytz-2021.1-5.el9.noarch 15/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-toml-0.10.2-6.el9.noarch 16/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-bcrypt-3.2.2-1.el9.x86_64 17/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cheroot-8.6.0-4.el9.noarch 18/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cherrypy-18.6.1-2.el9.noarch 19/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-8.2.1-3.el9.noarch 20/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-classes-3.2.1-5.el9.noarch 21/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-collections-3.0.0-8.el9.noarch 22/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-functools-3.5.0-2.el9.noarch 23/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jaraco-text-3.2.0-6.el9.noarch 24/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-logutils-0.3.5-21.el9.noarch 25/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-more-itertools-8.12.0-2.el9.noarch 26/33 2024-06-15T00:45:40.811 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pecan-1.4.2-3.el9.noarch 27/33 2024-06-15T00:45:40.812 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-portend-3.1.0-2.el9.noarch 28/33 2024-06-15T00:45:40.812 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyOpenSSL-21.0.0-1.el9.noarch 29/33 2024-06-15T00:45:40.812 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-tempora-5.0.0-2.el9.noarch 30/33 2024-06-15T00:45:40.812 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-webob-1.8.7-6.el9.noarch 31/33 2024-06-15T00:45:40.812 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-werkzeug-2.0.3-3.el9.1.noarch 32/33 2024-06-15T00:45:41.334 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-15T00:45:41.334 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:41.334 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:45:41.334 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-15T00:45:41.335 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-ply-3.11-14.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-15T00:45:41.336 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:41.337 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:41.664 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-base 2024-06-15T00:45:41.761 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-zc-lockfile-2.0-10.el9.noarch 33/33 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mds-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-modules-core-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mon-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-osd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: python3-bcrypt-3.2.2-1.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cffi-1.14.5-5.el9.x86_64 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cheroot-8.6.0-4.el9.noarch 2024-06-15T00:45:41.762 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cherrypy-18.6.1-2.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cryptography-36.0.1-4.el9.x86_64 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-8.2.1-3.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-classes-3.2.1-5.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-collections-3.0.0-8.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-functools-3.5.0-2.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jaraco-text-3.2.0-6.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-logutils-0.3.5-21.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-mako-1.1.4-6.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-markupsafe-1.1.1-12.el9.x86_64 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-more-itertools-8.12.0-2.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pecan-1.4.2-3.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-ply-3.11-14.el9.noarch 2024-06-15T00:45:41.763 INFO:teuthology.orchestra.run.smithi143.stdout: python3-portend-3.1.0-2.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyOpenSSL-21.0.0-1.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pycparser-2.20-6.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pytz-2021.1-5.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-2.25.1-8.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-tempora-5.0.0-2.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-toml-0.10.2-6.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-urllib3-1.26.5-5.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-webob-1.8.7-6.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-werkzeug-2.0.3-3.el9.1.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: python3-zc-lockfile-2.0-10.el9.noarch 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:41.764 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:42.116 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-base 2024-06-15T00:45:42.274 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:52 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:42.358 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:45:42.400 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:42.401 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:45:42.401 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:42.471 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install cephadm 2024-06-15T00:45:42.709 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:56 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:42.794 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-base-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:45:42.836 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:42.837 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:45:42.837 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:42.907 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install cephadm 2024-06-15T00:45:43.051 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:53 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:43.178 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:43.178 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:43.179 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:45:43.179 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:43.179 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:45:43.179 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 82 k 2024-06-15T00:45:43.179 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:43.179 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:45:43.180 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:43.180 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:45:43.180 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:43.180 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 82 k 2024-06-15T00:45:43.180 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 372 k 2024-06-15T00:45:43.180 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:45:43.484 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:57 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:43.499 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm-17.2.7-1008.ge47ff4b8.el9.noarch.rpm 256 kB/s | 82 kB 00:00 2024-06-15T00:45:43.499 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:43.500 INFO:teuthology.orchestra.run.smithi105.stdout:Total 255 kB/s | 82 kB 00:00 2024-06-15T00:45:43.500 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:45:43.505 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:45:43.505 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:45:43.507 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:45:43.508 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:45:43.580 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:45:43.610 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:43.611 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:43.611 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 82 k 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 82 k 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 372 k 2024-06-15T00:45:43.612 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:45:43.818 INFO:teuthology.orchestra.run.smithi143.stdout:cephadm-17.2.7-1008.ge47ff4b8.el9.noarch.rpm 399 kB/s | 82 kB 00:00 2024-06-15T00:45:43.819 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:43.819 INFO:teuthology.orchestra.run.smithi143.stdout:Total 396 kB/s | 82 kB 00:00 2024-06-15T00:45:43.819 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:45:43.824 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:45:43.824 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:45:43.826 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:45:43.826 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:45:43.897 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:45:44.023 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.118 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.342 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.430 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.505 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.753 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.753 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:44.753 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:45:44.753 INFO:teuthology.orchestra.run.smithi105.stdout: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:44.753 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:44.753 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:44.777 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:44.979 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-immutable-object-cache 2024-06-15T00:45:45.026 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/1 2024-06-15T00:45:45.026 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:45.026 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:45:45.026 INFO:teuthology.orchestra.run.smithi143.stdout: cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:45.026 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:45.027 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:45.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-immutable-object-cache 2024-06-15T00:45:45.565 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:55 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:45.695 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:45.696 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:45.696 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-06-15T00:45:45.696 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:45.696 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 137 k 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 137 k 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 379 k 2024-06-15T00:45:45.697 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:45:45.829 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:00:59 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:45.959 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 137 k 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:45:45.960 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:45.961 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:45:45.961 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:45.961 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 137 k 2024-06-15T00:45:45.961 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 379 k 2024-06-15T00:45:45.961 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:45:46.038 INFO:teuthology.orchestra.run.smithi105.stdout:ceph-immutable-object-cache-17.2.7-1008.ge47ff4 403 kB/s | 137 kB 00:00 2024-06-15T00:45:46.038 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:46.039 INFO:teuthology.orchestra.run.smithi105.stdout:Total 401 kB/s | 137 kB 00:00 2024-06-15T00:45:46.039 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:45:46.047 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:45:46.047 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:45:46.103 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:45:46.104 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:45:46.250 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:45:46.273 INFO:teuthology.orchestra.run.smithi143.stdout:ceph-immutable-object-cache-17.2.7-1008.ge47ff4 440 kB/s | 137 kB 00:00 2024-06-15T00:45:46.274 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:46.274 INFO:teuthology.orchestra.run.smithi143.stdout:Total 438 kB/s | 137 kB 00:00 2024-06-15T00:45:46.274 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:45:46.282 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:45:46.282 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:45:46.338 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:45:46.339 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-15T00:45:46.339 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:45:46.364 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-15T00:45:46.364 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:46.364 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-15T00:45:46.364 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:46.489 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:45:46.575 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-15T00:45:46.600 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-15T00:45:46.600 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:45:46.600 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-immutable-object-cache@*.service" escaped as "ceph-immutable-object-cache@\x2a.service". 2024-06-15T00:45:46.600 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-15T00:45:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:47.278 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:45:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:47.278 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:47.278 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:47.517 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr 2024-06-15T00:45:47.518 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8. 1/1 2024-06-15T00:45:47.518 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:47.518 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:45:47.518 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-immutable-object-cache-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:45:47.518 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:47.518 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:47.701 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr 2024-06-15T00:45:48.094 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:58 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:48.179 INFO:teuthology.orchestra.run.smithi105.stdout:Package ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:45:48.220 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:48.221 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:45:48.221 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:48.290 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-dashboard 2024-06-15T00:45:48.293 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:02 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:48.378 INFO:teuthology.orchestra.run.smithi143.stdout:Package ceph-mgr-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:45:48.420 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:48.421 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:45:48.421 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:48.487 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-dashboard 2024-06-15T00:45:48.855 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:00:58 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:48.981 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:48.981 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 3.4 M 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 24 k 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 15 k 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-15T00:45:48.982 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout:Installing weak dependencies: 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout:Install 7 Packages 2024-06-15T00:45:48.983 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:48.984 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 3.7 M 2024-06-15T00:45:48.984 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 80 M 2024-06-15T00:45:48.984 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:45:49.061 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:03 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:49.187 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 3.4 M 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:45:49.188 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 24 k 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 15 k 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt noarch 2.4.0-1.el9 epel 41 k 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru noarch 0.7-16.el9 epel 31 k 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes noarch 2.5.1-5.el9 epel 188 k 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout:Installing weak dependencies: 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt+crypto noarch 2.4.0-1.el9 epel 9.0 k 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout:Install 7 Packages 2024-06-15T00:45:49.189 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:49.190 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 3.7 M 2024-06-15T00:45:49.190 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 80 M 2024-06-15T00:45:49.190 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:45:49.325 INFO:teuthology.orchestra.run.smithi105.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1008.ge47 98 kB/s | 24 kB 00:00 2024-06-15T00:45:49.376 INFO:teuthology.orchestra.run.smithi105.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1008.ge47f 49 kB/s | 15 kB 00:00 2024-06-15T00:45:49.402 INFO:teuthology.orchestra.run.smithi105.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 120 kB/s | 9.0 kB 00:00 2024-06-15T00:45:49.451 INFO:teuthology.orchestra.run.smithi105.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.1 MB/s | 31 kB 00:00 2024-06-15T00:45:49.476 INFO:teuthology.orchestra.run.smithi105.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 415 kB/s | 41 kB 00:00 2024-06-15T00:45:49.501 INFO:teuthology.orchestra.run.smithi105.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.9 MB/s | 188 kB 00:00 2024-06-15T00:45:49.694 INFO:teuthology.orchestra.run.smithi105.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1008.ge47ff4b8 5.4 MB/s | 3.4 MB 00:00 2024-06-15T00:45:49.697 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:49.697 INFO:teuthology.orchestra.run.smithi105.stdout:Total 5.1 MB/s | 3.7 MB 00:00 2024-06-15T00:45:49.697 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:45:49.705 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:45:49.705 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:45:49.767 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:45:49.767 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:45:49.810 INFO:teuthology.orchestra.run.smithi143.stdout:(1/7): ceph-grafana-dashboards-17.2.7-1008.ge47 127 kB/s | 24 kB 00:00 2024-06-15T00:45:49.835 INFO:teuthology.orchestra.run.smithi143.stdout:(2/7): ceph-prometheus-alerts-17.2.7-1008.ge47f 67 kB/s | 15 kB 00:00 2024-06-15T00:45:49.918 INFO:teuthology.orchestra.run.smithi143.stdout:(3/7): python3-jwt+crypto-2.4.0-1.el9.noarch.rp 83 kB/s | 9.0 kB 00:00 2024-06-15T00:45:49.944 INFO:teuthology.orchestra.run.smithi143.stdout:(4/7): python3-repoze-lru-0.7-16.el9.noarch.rpm 1.3 MB/s | 31 kB 00:00 2024-06-15T00:45:49.969 INFO:teuthology.orchestra.run.smithi143.stdout:(5/7): python3-jwt-2.4.0-1.el9.noarch.rpm 305 kB/s | 41 kB 00:00 2024-06-15T00:45:49.994 INFO:teuthology.orchestra.run.smithi143.stdout:(6/7): python3-routes-2.5.1-5.el9.noarch.rpm 3.7 MB/s | 188 kB 00:00 2024-06-15T00:45:50.079 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:45:50.136 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-15T00:45:50.162 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-15T00:45:50.208 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-15T00:45:50.211 INFO:teuthology.orchestra.run.smithi143.stdout:(7/7): ceph-mgr-dashboard-17.2.7-1008.ge47ff4b8 5.7 MB/s | 3.4 MB 00:00 2024-06-15T00:45:50.217 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:50.218 INFO:teuthology.orchestra.run.smithi143.stdout:Total 3.6 MB/s | 3.7 MB 00:01 2024-06-15T00:45:50.218 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:45:50.223 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:45:50.223 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:45:50.249 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-15T00:45:50.269 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 5/7 2024-06-15T00:45:50.287 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:45:50.287 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:45:50.591 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:45:50.654 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jwt-2.4.0-1.el9.noarch 1/7 2024-06-15T00:45:50.688 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jwt+crypto-2.4.0-1.el9.noarch 2/7 2024-06-15T00:45:50.734 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-repoze-lru-0.7-16.el9.noarch 3/7 2024-06-15T00:45:50.766 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-routes-2.5.1-5.el9.noarch 4/7 2024-06-15T00:45:50.787 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 5/7 2024-06-15T00:45:51.244 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 6/7 2024-06-15T00:45:51.290 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-15T00:45:51.777 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 6/7 2024-06-15T00:45:51.816 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 1/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 2/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 3/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-15T00:45:52.302 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-15T00:45:52.635 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-15T00:45:52.635 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:52.635 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:45:52.635 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:52.636 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:45:52.866 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 7/7 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9. 1/7 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarc 2/7 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.n 3/7 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jwt+crypto-2.4.0-1.el9.noarch 4/7 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jwt-2.4.0-1.el9.noarch 5/7 2024-06-15T00:45:52.885 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-repoze-lru-0.7-16.el9.noarch 6/7 2024-06-15T00:45:53.208 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-routes-2.5.1-5.el9.noarch 7/7 2024-06-15T00:45:53.208 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-grafana-dashboards-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-dashboard-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-prometheus-alerts-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt-2.4.0-1.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jwt+crypto-2.4.0-1.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: python3-repoze-lru-0.7-16.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: python3-routes-2.5.1-5.el9.noarch 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:53.209 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:45:53.409 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-diskprediction-local 2024-06-15T00:45:53.437 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:03 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:45:53.565 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:45:53.566 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-06-15T00:45:53.566 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 7.4 M 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-15T00:45:53.567 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:======================================================================================= 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout:Install 12 Packages 2024-06-15T00:45:53.568 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:45:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 38 M 2024-06-15T00:45:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 209 M 2024-06-15T00:45:53.569 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:45:53.978 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:07 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:45:54.106 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 7.4 M 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas x86_64 3.0.4-8.el9 appstream 33 k 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib x86_64 3.0.4-8.el9 appstream 3.0 M 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp x86_64 3.0.4-8.el9 appstream 18 k 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran x86_64 11.4.1-3.el9 baseos 803 k 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath x86_64 11.4.1-3.el9 baseos 195 k 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: openblas x86_64 0.3.21-2.el9 appstream 35 k 2024-06-15T00:45:54.107 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp x86_64 0.3.21-2.el9 appstream 4.7 M 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel x86_64 3.9.18-3.el9 appstream 244 k 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy x86_64 1:1.20.1-5.el9 appstream 5.1 M 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py x86_64 1:1.20.1-5.el9 appstream 250 k 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy x86_64 1.6.2-8.el9 appstream 16 M 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:======================================================================================= 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:Install 12 Packages 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 38 M 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 209 M 2024-06-15T00:45:54.108 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:45:54.476 INFO:teuthology.orchestra.run.smithi105.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 304 kB/s | 195 kB 00:00 2024-06-15T00:45:54.485 INFO:teuthology.orchestra.run.smithi143.stdout:(1/12): libquadmath-11.4.1-3.el9.x86_64.rpm 1.3 MB/s | 195 kB 00:00 2024-06-15T00:45:54.544 INFO:teuthology.orchestra.run.smithi105.stdout:(2/12): flexiblas-3.0.4-8.el9.x86_64.rpm 491 kB/s | 33 kB 00:00 2024-06-15T00:45:54.544 INFO:teuthology.orchestra.run.smithi143.stdout:(2/12): libgfortran-11.4.1-3.el9.x86_64.rpm 3.9 MB/s | 803 kB 00:00 2024-06-15T00:45:54.636 INFO:teuthology.orchestra.run.smithi143.stdout:(3/12): flexiblas-3.0.4-8.el9.x86_64.rpm 219 kB/s | 33 kB 00:00 2024-06-15T00:45:54.686 INFO:teuthology.orchestra.run.smithi143.stdout:(4/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 363 kB/s | 18 kB 00:00 2024-06-15T00:45:54.693 INFO:teuthology.orchestra.run.smithi105.stdout:(3/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 20 MB/s | 3.0 MB 00:00 2024-06-15T00:45:54.719 INFO:teuthology.orchestra.run.smithi105.stdout:(4/12): libgfortran-11.4.1-3.el9.x86_64.rpm 907 kB/s | 803 kB 00:00 2024-06-15T00:45:54.736 INFO:teuthology.orchestra.run.smithi105.stdout:(5/12): flexiblas-openblas-openmp-3.0.4-8.el9.x 434 kB/s | 18 kB 00:00 2024-06-15T00:45:54.736 INFO:teuthology.orchestra.run.smithi143.stdout:(5/12): openblas-0.3.21-2.el9.x86_64.rpm 708 kB/s | 35 kB 00:00 2024-06-15T00:45:54.786 INFO:teuthology.orchestra.run.smithi105.stdout:(6/12): openblas-0.3.21-2.el9.x86_64.rpm 525 kB/s | 35 kB 00:00 2024-06-15T00:45:54.870 INFO:teuthology.orchestra.run.smithi105.stdout:(7/12): openblas-openmp-0.3.21-2.el9.x86_64.rpm 35 MB/s | 4.7 MB 00:00 2024-06-15T00:45:54.922 INFO:teuthology.orchestra.run.smithi105.stdout:(8/12): python3-devel-3.9.18-3.el9.x86_64.rpm 1.8 MB/s | 244 kB 00:00 2024-06-15T00:45:54.996 INFO:teuthology.orchestra.run.smithi105.stdout:(9/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 41 MB/s | 5.1 MB 00:00 2024-06-15T00:45:55.021 INFO:teuthology.orchestra.run.smithi105.stdout:(10/12): python3-numpy-f2py-1.20.1-5.el9.x86_64 2.4 MB/s | 250 kB 00:00 2024-06-15T00:45:55.203 INFO:teuthology.orchestra.run.smithi143.stdout:(6/12): ceph-mgr-diskprediction-local-17.2.7-10 8.6 MB/s | 7.4 MB 00:00 2024-06-15T00:45:55.296 INFO:teuthology.orchestra.run.smithi105.stdout:(11/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 54 MB/s | 16 MB 00:00 2024-06-15T00:45:55.462 INFO:teuthology.orchestra.run.smithi143.stdout:(7/12): python3-devel-3.9.18-3.el9.x86_64.rpm 946 kB/s | 244 kB 00:00 2024-06-15T00:45:55.530 INFO:teuthology.orchestra.run.smithi105.stdout:(12/12): ceph-mgr-diskprediction-local-17.2.7-1 4.4 MB/s | 7.4 MB 00:01 2024-06-15T00:45:55.531 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:45:55.532 INFO:teuthology.orchestra.run.smithi105.stdout:Total 19 MB/s | 38 MB 00:01 2024-06-15T00:45:55.804 INFO:teuthology.orchestra.run.smithi143.stdout:(8/12): flexiblas-netlib-3.0.4-8.el9.x86_64.rpm 2.4 MB/s | 3.0 MB 00:01 2024-06-15T00:45:55.884 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:45:55.910 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:45:55.910 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:45:55.979 INFO:teuthology.orchestra.run.smithi143.stdout:(9/12): python3-numpy-f2py-1.20.1-5.el9.x86_64. 1.4 MB/s | 250 kB 00:00 2024-06-15T00:45:56.196 INFO:teuthology.orchestra.run.smithi143.stdout:(10/12): python3-numpy-1.20.1-5.el9.x86_64.rpm 6.9 MB/s | 5.1 MB 00:00 2024-06-15T00:45:56.423 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:45:56.424 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:45:56.712 INFO:teuthology.orchestra.run.smithi143.stdout:(11/12): openblas-openmp-0.3.21-2.el9.x86_64.rp 2.4 MB/s | 4.7 MB 00:01 2024-06-15T00:45:57.191 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:45:57.264 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-15T00:45:57.302 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-15T00:45:57.406 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-15T00:45:57.446 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-15T00:45:57.630 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-15T00:45:57.669 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-15T00:45:57.755 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-15T00:45:58.413 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-15T00:45:58.542 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-15T00:45:59.771 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-15T00:46:00.804 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-15T00:46:00.836 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-15T00:46:03.448 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-15T00:46:03.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 1/12 2024-06-15T00:46:03.448 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-15T00:46:03.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-15T00:46:03.450 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-15T00:46:03.853 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-15T00:46:03.853 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-15T00:46:03.854 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:03.855 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:04.258 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-rook 2024-06-15T00:46:04.852 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:14 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:04.978 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 47 k 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-15T00:46:04.982 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:04.983 INFO:teuthology.orchestra.run.smithi105.stdout:Install 13 Packages 2024-06-15T00:46:04.984 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:04.984 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.2 M 2024-06-15T00:46:04.984 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 26 M 2024-06-15T00:46:04.984 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:05.420 INFO:teuthology.orchestra.run.smithi143.stdout:(12/12): python3-scipy-1.6.2-8.el9.x86_64.rpm 1.7 MB/s | 16 MB 00:09 2024-06-15T00:46:05.421 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:05.421 INFO:teuthology.orchestra.run.smithi143.stdout:Total 3.4 MB/s | 38 MB 00:11 2024-06-15T00:46:05.604 INFO:teuthology.orchestra.run.smithi105.stdout:(1/13): ceph-mgr-rook-17.2.7-1008.ge47ff4b8.el9 259 kB/s | 47 kB 00:00 2024-06-15T00:46:05.696 INFO:teuthology.orchestra.run.smithi105.stdout:(2/13): python3-jsonpatch-1.21-16.el9.noarch.rp 93 kB/s | 26 kB 00:00 2024-06-15T00:46:05.721 INFO:teuthology.orchestra.run.smithi105.stdout:(3/13): python3-jsonpointer-2.0-4.el9.noarch.rp 64 kB/s | 19 kB 00:00 2024-06-15T00:46:05.779 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:05.804 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:05.805 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:06.071 INFO:teuthology.orchestra.run.smithi105.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 454 kB/s | 159 kB 00:00 2024-06-15T00:46:06.146 INFO:teuthology.orchestra.run.smithi105.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 410 kB/s | 222 kB 00:00 2024-06-15T00:46:06.196 INFO:teuthology.orchestra.run.smithi105.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 428 kB/s | 54 kB 00:00 2024-06-15T00:46:06.222 INFO:teuthology.orchestra.run.smithi105.stdout:(7/13): python3-pyasn1-modules-0.4.8-6.el9.noar 557 kB/s | 279 kB 00:00 2024-06-15T00:46:06.272 INFO:teuthology.orchestra.run.smithi105.stdout:(8/13): python3-cachetools-4.2.4-1.el9.noarch.r 256 kB/s | 32 kB 00:00 2024-06-15T00:46:06.309 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:06.309 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:06.322 INFO:teuthology.orchestra.run.smithi105.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 121 kB/s | 15 kB 00:00 2024-06-15T00:46:06.347 INFO:teuthology.orchestra.run.smithi105.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.5 MB/s | 59 kB 00:00 2024-06-15T00:46:06.381 INFO:teuthology.orchestra.run.smithi105.stdout:(11/13): python3-websocket-client-1.2.3-2.el9.n 2.6 MB/s | 90 kB 00:00 2024-06-15T00:46:06.407 INFO:teuthology.orchestra.run.smithi105.stdout:(12/13): python3-google-auth-2.29.0-2.el9.noarc 1.2 MB/s | 205 kB 00:00 2024-06-15T00:46:06.440 INFO:teuthology.orchestra.run.smithi105.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 7.2 MB/s | 1.0 MB 00:00 2024-06-15T00:46:06.442 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:06.443 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.5 MB/s | 2.2 MB 00:01 2024-06-15T00:46:06.505 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:06.519 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:06.519 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:06.604 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:06.606 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:07.070 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:07.137 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:07.149 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-3.0.4-8.el9.x86_64 1/12 2024-06-15T00:46:07.201 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libquadmath-11.4.1-3.el9.x86_64 2/12 2024-06-15T00:46:07.301 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libgfortran-11.4.1-3.el9.x86_64 3/12 2024-06-15T00:46:07.328 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-15T00:46:07.345 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-devel-3.9.18-3.el9.x86_64 4/12 2024-06-15T00:46:07.393 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-15T00:46:07.456 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-15T00:46:07.502 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-15T00:46:07.532 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-0.3.21-2.el9.x86_64 5/12 2024-06-15T00:46:07.552 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-15T00:46:07.568 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : openblas-openmp-0.3.21-2.el9.x86_64 6/12 2024-06-15T00:46:07.655 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-15T00:46:07.657 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 7/12 2024-06-15T00:46:07.763 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-15T00:46:07.812 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-15T00:46:08.291 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : flexiblas-netlib-3.0.4-8.el9.x86_64 8/12 2024-06-15T00:46:08.412 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-1:1.20.1-5.el9.x86_64 9/12 2024-06-15T00:46:08.519 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-15T00:46:08.576 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-15T00:46:08.610 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-15T00:46:08.655 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-15T00:46:08.679 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-15T00:46:09.436 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-15T00:46:09.437 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-15T00:46:09.437 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-15T00:46:09.437 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-15T00:46:09.437 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-15T00:46:09.437 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-15T00:46:09.623 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 10/12 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-15T00:46:09.766 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:09.767 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:10.034 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-mgr-cephadm 2024-06-15T00:46:10.604 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:20 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:10.666 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-scipy-1.6.2-8.el9.x86_64 11/12 2024-06-15T00:46:10.699 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-15T00:46:10.729 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 123 k 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-15T00:46:10.730 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-15T00:46:10.731 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-15T00:46:10.731 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:10.731 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:10.731 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:10.733 INFO:teuthology.orchestra.run.smithi105.stdout:Install 6 Packages 2024-06-15T00:46:10.733 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:10.734 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 7.0 M 2024-06-15T00:46:10.734 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 33 M 2024-06-15T00:46:10.734 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:11.068 INFO:teuthology.orchestra.run.smithi105.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.1 MB/s | 248 kB 00:00 2024-06-15T00:46:11.143 INFO:teuthology.orchestra.run.smithi105.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 7.1 MB/s | 548 kB 00:00 2024-06-15T00:46:11.169 INFO:teuthology.orchestra.run.smithi105.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.2 MB/s | 58 kB 00:00 2024-06-15T00:46:11.202 INFO:teuthology.orchestra.run.smithi105.stdout:(4/6): ceph-mgr-cephadm-17.2.7-1008.ge47ff4b8.e 349 kB/s | 123 kB 00:00 2024-06-15T00:46:11.228 INFO:teuthology.orchestra.run.smithi105.stdout:(5/6): python3-typing-extensions-4.4.0-2.el9.no 870 kB/s | 51 kB 00:00 2024-06-15T00:46:11.403 INFO:teuthology.orchestra.run.smithi105.stdout:(6/6): python3-babel-2.9.1-2.el9.noarch.rpm 11 MB/s | 6.0 MB 00:00 2024-06-15T00:46:11.405 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:11.405 INFO:teuthology.orchestra.run.smithi105.stdout:Total 10 MB/s | 7.0 MB 00:00 2024-06-15T00:46:11.474 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:11.485 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:11.485 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:11.581 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:11.581 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:11.889 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:12.054 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-15T00:46:12.105 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-15T00:46:12.632 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-15T00:46:12.718 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-15T00:46:12.772 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-15T00:46:12.801 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 12/12 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff 1/12 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libgfortran-11.4.1-3.el9.x86_64 2/12 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libquadmath-11.4.1-3.el9.x86_64 3/12 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-3.0.4-8.el9.x86_64 4/12 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-netlib-3.0.4-8.el9.x86_64 5/12 2024-06-15T00:46:13.402 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 6/12 2024-06-15T00:46:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-0.3.21-2.el9.x86_64 7/12 2024-06-15T00:46:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : openblas-openmp-0.3.21-2.el9.x86_64 8/12 2024-06-15T00:46:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-devel-3.9.18-3.el9.x86_64 9/12 2024-06-15T00:46:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-1:1.20.1-5.el9.x86_64 10/12 2024-06-15T00:46:13.403 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-numpy-f2py-1:1.20.1-5.el9.x86_64 11/12 2024-06-15T00:46:13.537 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-15T00:46:13.537 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/6 2024-06-15T00:46:13.537 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-15T00:46:13.537 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-15T00:46:13.537 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-15T00:46:13.537 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-15T00:46:13.806 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-15T00:46:13.806 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:13.807 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-scipy-1.6.2-8.el9.x86_64 12/12 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-diskprediction-local-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-3.0.4-8.el9.x86_64 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-netlib-3.0.4-8.el9.x86_64 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: flexiblas-openblas-openmp-3.0.4-8.el9.x86_64 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: libgfortran-11.4.1-3.el9.x86_64 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: libquadmath-11.4.1-3.el9.x86_64 2024-06-15T00:46:13.816 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-0.3.21-2.el9.x86_64 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: openblas-openmp-0.3.21-2.el9.x86_64 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-devel-3.9.18-3.el9.x86_64 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-1:1.20.1-5.el9.x86_64 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-numpy-f2py-1:1.20.1-5.el9.x86_64 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: python3-scipy-1.6.2-8.el9.x86_64 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:13.817 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:14.042 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install ceph-fuse 2024-06-15T00:46:14.246 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-rook 2024-06-15T00:46:14.614 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:24 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:14.740 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:14.741 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:14.741 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repository Size 2024-06-15T00:46:14.741 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:14.741 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:14.741 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 812 k 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout:Installing dependencies: 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout:Install 2 Packages 2024-06-15T00:46:14.742 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:14.743 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 892 k 2024-06-15T00:46:14.743 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 2.6 M 2024-06-15T00:46:14.743 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:14.834 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:28 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:14.962 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:14.963 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:14.963 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:46:14.963 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:14.963 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:14.963 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 47 k 2024-06-15T00:46:14.963 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools noarch 4.2.4-1.el9 epel 32 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi noarch 2021.10.8-2.el9 epel 15 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth noarch 1:2.29.0-2.el9 epel 205 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch noarch 1.21-16.el9 appstream 26 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer noarch 2.0-4.el9 appstream 19 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes noarch 1:26.1.0-2.el9 epel 1.0 M 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib noarch 3.1.1-5.el9 appstream 222 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1 noarch 0.4.8-6.el9 appstream 159 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules noarch 0.4.8-6.el9 appstream 279 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib noarch 1.3.0-12.el9 appstream 54 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa noarch 4.9-2.el9 epel 59 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client noarch 1.2.3-2.el9 epel 90 k 2024-06-15T00:46:14.964 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout:Install 13 Packages 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 2.2 M 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 26 M 2024-06-15T00:46:14.965 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:14.980 INFO:teuthology.orchestra.run.smithi105.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 672 kB/s | 80 kB 00:00 2024-06-15T00:46:15.297 INFO:teuthology.orchestra.run.smithi105.stdout:(2/2): ceph-fuse-17.2.7-1008.ge47ff4b8.el9.x86_ 1.8 MB/s | 812 kB 00:00 2024-06-15T00:46:15.298 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:15.298 INFO:teuthology.orchestra.run.smithi105.stdout:Total 1.6 MB/s | 892 kB 00:00 2024-06-15T00:46:15.310 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:15.332 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:15.332 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:15.410 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:15.410 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:15.615 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:15.683 INFO:teuthology.orchestra.run.smithi143.stdout:(1/13): python3-jsonpatch-1.21-16.el9.noarch.rp 174 kB/s | 26 kB 00:00 2024-06-15T00:46:15.700 INFO:teuthology.orchestra.run.smithi143.stdout:(2/13): python3-jsonpointer-2.0-4.el9.noarch.rp 117 kB/s | 19 kB 00:00 2024-06-15T00:46:15.728 INFO:teuthology.orchestra.run.smithi143.stdout:(3/13): ceph-mgr-rook-17.2.7-1008.ge47ff4b8.el9 246 kB/s | 47 kB 00:00 2024-06-15T00:46:15.744 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-15T00:46:15.812 INFO:teuthology.orchestra.run.smithi143.stdout:(4/13): python3-pyasn1-0.4.8-6.el9.noarch.rpm 1.4 MB/s | 159 kB 00:00 2024-06-15T00:46:15.813 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-15T00:46:15.837 INFO:teuthology.orchestra.run.smithi143.stdout:(5/13): python3-oauthlib-3.1.1-5.el9.noarch.rpm 1.6 MB/s | 222 kB 00:00 2024-06-15T00:46:15.887 INFO:teuthology.orchestra.run.smithi143.stdout:(6/13): python3-requests-oauthlib-1.3.0-12.el9. 713 kB/s | 54 kB 00:00 2024-06-15T00:46:15.963 INFO:teuthology.orchestra.run.smithi143.stdout:(7/13): python3-cachetools-4.2.4-1.el9.noarch.r 257 kB/s | 32 kB 00:00 2024-06-15T00:46:15.996 INFO:teuthology.orchestra.run.smithi143.stdout:(8/13): python3-pyasn1-modules-0.4.8-6.el9.noar 1.0 MB/s | 279 kB 00:00 2024-06-15T00:46:16.022 INFO:teuthology.orchestra.run.smithi143.stdout:(9/13): python3-certifi-2021.10.8-2.el9.noarch. 114 kB/s | 15 kB 00:00 2024-06-15T00:46:16.072 INFO:teuthology.orchestra.run.smithi143.stdout:(10/13): python3-rsa-4.9-2.el9.noarch.rpm 2.2 MB/s | 59 kB 00:00 2024-06-15T00:46:16.097 INFO:teuthology.orchestra.run.smithi143.stdout:(11/13): python3-google-auth-2.29.0-2.el9.noarc 1.5 MB/s | 205 kB 00:00 2024-06-15T00:46:16.122 INFO:teuthology.orchestra.run.smithi143.stdout:(12/13): python3-websocket-client-1.2.3-2.el9.n 1.8 MB/s | 90 kB 00:00 2024-06-15T00:46:16.164 INFO:teuthology.orchestra.run.smithi143.stdout:(13/13): python3-kubernetes-26.1.0-2.el9.noarch 6.1 MB/s | 1.0 MB 00:00 2024-06-15T00:46:16.166 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:16.167 INFO:teuthology.orchestra.run.smithi143.stdout:Total 1.8 MB/s | 2.2 MB 00:01 2024-06-15T00:46:16.231 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:16.246 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:16.246 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:16.331 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:16.332 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:16.456 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-15T00:46:16.456 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/2 2024-06-15T00:46:16.707 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-15T00:46:16.707 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:16.708 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:16.708 INFO:teuthology.orchestra.run.smithi105.stdout: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-15T00:46:16.708 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:16.708 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:16.834 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:16.961 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados-devel 2024-06-15T00:46:17.017 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-0.4.8-6.el9.noarch 1/13 2024-06-15T00:46:17.073 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-pyasn1-modules-0.4.8-6.el9.noarch 2/13 2024-06-15T00:46:17.129 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-rsa-4.9-2.el9.noarch 3/13 2024-06-15T00:46:17.166 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-websocket-client-1.2.3-2.el9.noarch 4/13 2024-06-15T00:46:17.208 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-certifi-2021.10.8-2.el9.noarch 5/13 2024-06-15T00:46:17.318 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-cachetools-4.2.4-1.el9.noarch 6/13 2024-06-15T00:46:17.418 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-google-auth-1:2.29.0-2.el9.noarch 7/13 2024-06-15T00:46:17.476 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-oauthlib-3.1.1-5.el9.noarch 8/13 2024-06-15T00:46:17.549 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:27 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:17.677 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:17.677 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:17.677 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 126 k 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 126 k 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 448 k 2024-06-15T00:46:17.678 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:17.994 INFO:teuthology.orchestra.run.smithi105.stdout:librados-devel-17.2.7-1008.ge47ff4b8.el9.x86_64 400 kB/s | 126 kB 00:00 2024-06-15T00:46:17.995 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:17.995 INFO:teuthology.orchestra.run.smithi105.stdout:Total 398 kB/s | 126 kB 00:00 2024-06-15T00:46:17.995 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:18.000 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:18.000 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:18.063 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:18.064 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:18.182 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-requests-oauthlib-1.3.0-12.el9.noarch 9/13 2024-06-15T00:46:18.217 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:18.240 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-kubernetes-1:26.1.0-2.el9.noarch 10/13 2024-06-15T00:46:18.274 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpointer-2.0-4.el9.noarch 11/13 2024-06-15T00:46:18.308 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:18.320 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jsonpatch-1.21-16.el9.noarch 12/13 2024-06-15T00:46:18.344 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-15T00:46:18.723 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:18.997 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:18.997 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:18.997 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:18.997 INFO:teuthology.orchestra.run.smithi105.stdout: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:18.997 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:18.997 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:19.077 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 13/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpatch-1.21-16.el9.noarch 2/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jsonpointer-2.0-4.el9.noarch 3/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-oauthlib-3.1.1-5.el9.noarch 4/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-0.4.8-6.el9.noarch 5/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-pyasn1-modules-0.4.8-6.el9.noarch 6/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-requests-oauthlib-1.3.0-12.el9.noarch 7/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-cachetools-4.2.4-1.el9.noarch 8/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-certifi-2021.10.8-2.el9.noarch 9/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-google-auth-1:2.29.0-2.el9.noarch 10/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-kubernetes-1:26.1.0-2.el9.noarch 11/13 2024-06-15T00:46:19.078 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-rsa-4.9-2.el9.noarch 12/13 2024-06-15T00:46:19.203 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs2 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-websocket-client-1.2.3-2.el9.noarch 13/13 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-rook-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: python3-cachetools-4.2.4-1.el9.noarch 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: python3-certifi-2021.10.8-2.el9.noarch 2024-06-15T00:46:19.404 INFO:teuthology.orchestra.run.smithi143.stdout: python3-google-auth-1:2.29.0-2.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpatch-1.21-16.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jsonpointer-2.0-4.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-kubernetes-1:26.1.0-2.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-oauthlib-3.1.1-5.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-0.4.8-6.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-pyasn1-modules-0.4.8-6.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-requests-oauthlib-1.3.0-12.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-rsa-4.9-2.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: python3-websocket-client-1.2.3-2.el9.noarch 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:19.405 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:19.650 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-mgr-cephadm 2024-06-15T00:46:19.791 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:29 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:19.875 INFO:teuthology.orchestra.run.smithi105.stdout:Package libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:19.918 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:19.919 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:19.919 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:20.002 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install libcephfs-devel 2024-06-15T00:46:20.227 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:34 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:20.353 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm noarch 2:17.2.7-1008.ge47ff4b8.el9 ceph-noarch 123 k 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh noarch 2.13.2-5.el9 epel 548 k 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel noarch 2.9.1-2.el9 appstream 6.0 M 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2 noarch 2.11.3-6.el9 appstream 248 k 2024-06-15T00:46:20.354 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort noarch 7.1.1-5.el9 epel 58 k 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions noarch 4.4.0-2.el9 epel 51 k 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout:Install 6 Packages 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:20.355 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 7.0 M 2024-06-15T00:46:20.356 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 33 M 2024-06-15T00:46:20.356 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:20.578 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:30 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:20.701 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:20.702 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:20.702 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-06-15T00:46:20.702 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:20.702 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 25 k 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 110 k 2024-06-15T00:46:20.703 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:20.775 INFO:teuthology.orchestra.run.smithi143.stdout:(1/6): python3-jinja2-2.11.3-6.el9.noarch.rpm 1.8 MB/s | 248 kB 00:00 2024-06-15T00:46:20.842 INFO:teuthology.orchestra.run.smithi143.stdout:(2/6): python3-asyncssh-2.13.2-5.el9.noarch.rpm 8.0 MB/s | 548 kB 00:00 2024-06-15T00:46:20.865 INFO:teuthology.orchestra.run.smithi105.stdout:libcephfs-devel-17.2.7-1008.ge47ff4b8.el9.x86_6 154 kB/s | 25 kB 00:00 2024-06-15T00:46:20.865 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:20.866 INFO:teuthology.orchestra.run.smithi105.stdout:Total 153 kB/s | 25 kB 00:00 2024-06-15T00:46:20.866 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:20.867 INFO:teuthology.orchestra.run.smithi143.stdout:(3/6): python3-natsort-7.1.1-5.el9.noarch.rpm 2.3 MB/s | 58 kB 00:00 2024-06-15T00:46:20.869 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:20.869 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:20.870 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:20.871 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:20.893 INFO:teuthology.orchestra.run.smithi143.stdout:(4/6): python3-typing-extensions-4.4.0-2.el9.no 2.0 MB/s | 51 kB 00:00 2024-06-15T00:46:20.947 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:20.985 INFO:teuthology.orchestra.run.smithi143.stdout:(5/6): python3-babel-2.9.1-2.el9.noarch.rpm 17 MB/s | 6.0 MB 00:00 2024-06-15T00:46:21.010 INFO:teuthology.orchestra.run.smithi143.stdout:(6/6): ceph-mgr-cephadm-17.2.7-1008.ge47ff4b8.e 335 kB/s | 123 kB 00:00 2024-06-15T00:46:21.012 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:21.012 INFO:teuthology.orchestra.run.smithi143.stdout:Total 11 MB/s | 7.0 MB 00:00 2024-06-15T00:46:21.052 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:21.083 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:21.094 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:21.094 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:21.191 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:21.193 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:21.419 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:21.495 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:21.646 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:21.647 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:21.647 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:21.647 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:21.665 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-typing-extensions-4.4.0-2.el9.noarch 1/6 2024-06-15T00:46:21.712 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-asyncssh-2.13.2-5.el9.noarch 2/6 2024-06-15T00:46:21.833 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librados2 2024-06-15T00:46:22.215 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-natsort-7.1.1-5.el9.noarch 3/6 2024-06-15T00:46:22.306 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-babel-2.9.1-2.el9.noarch 4/6 2024-06-15T00:46:22.370 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : python3-jinja2-2.11.3-6.el9.noarch 5/6 2024-06-15T00:46:22.400 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-15T00:46:22.410 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:32 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:22.493 INFO:teuthology.orchestra.run.smithi105.stdout:Package librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:22.535 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:22.536 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:22.536 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:22.609 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install librbd1 2024-06-15T00:46:23.183 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:33 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:23.197 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 6/6 2024-06-15T00:46:23.197 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 1/6 2024-06-15T00:46:23.198 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-babel-2.9.1-2.el9.noarch 2/6 2024-06-15T00:46:23.198 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-jinja2-2.11.3-6.el9.noarch 3/6 2024-06-15T00:46:23.198 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-asyncssh-2.13.2-5.el9.noarch 4/6 2024-06-15T00:46:23.198 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-natsort-7.1.1-5.el9.noarch 5/6 2024-06-15T00:46:23.267 INFO:teuthology.orchestra.run.smithi105.stdout:Package librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:23.309 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:23.310 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:23.310 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:23.384 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rados 2024-06-15T00:46:23.469 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : python3-typing-extensions-4.4.0-2.el9.noarch 6/6 2024-06-15T00:46:23.469 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:23.469 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:23.469 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-mgr-cephadm-2:17.2.7-1008.ge47ff4b8.el9.noarch 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout: python3-asyncssh-2.13.2-5.el9.noarch 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout: python3-babel-2.9.1-2.el9.noarch 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout: python3-jinja2-2.11.3-6.el9.noarch 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout: python3-natsort-7.1.1-5.el9.noarch 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout: python3-typing-extensions-4.4.0-2.el9.noarch 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:23.470 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:23.723 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install ceph-fuse 2024-06-15T00:46:23.958 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:33 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:24.040 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:24.081 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:24.082 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:24.082 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:24.151 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rgw 2024-06-15T00:46:24.300 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:38 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:24.426 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:24.426 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repository Size 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 812 k 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:Installing dependencies: 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout: fuse x86_64 2.9.9-15.el9 baseos 80 k 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:Install 2 Packages 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 892 k 2024-06-15T00:46:24.427 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 2.6 M 2024-06-15T00:46:24.428 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:24.713 INFO:teuthology.orchestra.run.smithi143.stdout:(1/2): fuse-2.9.9-15.el9.x86_64.rpm 482 kB/s | 80 kB 00:00 2024-06-15T00:46:24.731 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:34 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:24.814 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:24.846 INFO:teuthology.orchestra.run.smithi143.stdout:(2/2): ceph-fuse-17.2.7-1008.ge47ff4b8.el9.x86_ 2.6 MB/s | 812 kB 00:00 2024-06-15T00:46:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:24.847 INFO:teuthology.orchestra.run.smithi143.stdout:Total 2.1 MB/s | 892 kB 00:00 2024-06-15T00:46:24.858 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:24.858 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:24.858 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:24.859 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:24.881 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:24.881 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:24.934 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-cephfs 2024-06-15T00:46:24.959 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:24.959 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:25.152 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:25.292 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : fuse-2.9.9-15.el9.x86_64 1/2 2024-06-15T00:46:25.354 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-15T00:46:25.501 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:35 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:25.584 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:25.625 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:25.626 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:25.626 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:25.696 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install python3-rbd 2024-06-15T00:46:25.947 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2/2 2024-06-15T00:46:25.947 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/2 2024-06-15T00:46:26.255 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : fuse-2.9.9-15.el9.x86_64 2/2 2024-06-15T00:46:26.255 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:26.255 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:26.255 INFO:teuthology.orchestra.run.smithi143.stdout: ceph-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 fuse-2.9.9-15.el9.x86_64 2024-06-15T00:46:26.255 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:26.255 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:26.272 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:36 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:26.356 INFO:teuthology.orchestra.run.smithi105.stdout:Package python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:26.399 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:26.400 INFO:teuthology.orchestra.run.smithi105.stdout:Nothing to do. 2024-06-15T00:46:26.400 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:26.481 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-fuse 2024-06-15T00:46:26.491 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados-devel 2024-06-15T00:46:27.056 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:37 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:27.088 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:41 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:27.181 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:27.182 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:27.182 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-06-15T00:46:27.182 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:27.182 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 82 k 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 82 k 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 222 k 2024-06-15T00:46:27.183 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:27.217 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 126 k 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:27.218 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:46:27.219 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:27.219 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 126 k 2024-06-15T00:46:27.219 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 448 k 2024-06-15T00:46:27.219 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:27.437 INFO:teuthology.orchestra.run.smithi143.stdout:librados-devel-17.2.7-1008.ge47ff4b8.el9.x86_64 583 kB/s | 126 kB 00:00 2024-06-15T00:46:27.438 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:27.439 INFO:teuthology.orchestra.run.smithi143.stdout:Total 573 kB/s | 126 kB 00:00 2024-06-15T00:46:27.439 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:27.445 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:27.445 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:27.493 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-fuse-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 264 kB/s | 82 kB 00:00 2024-06-15T00:46:27.494 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:27.494 INFO:teuthology.orchestra.run.smithi105.stdout:Total 263 kB/s | 82 kB 00:00 2024-06-15T00:46:27.494 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:27.501 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:27.501 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:27.509 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:27.509 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:27.563 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:27.563 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:27.667 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:27.702 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:27.782 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:27.799 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:28.161 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:28.166 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:28.411 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:28.412 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:28.412 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:28.412 INFO:teuthology.orchestra.run.smithi143.stdout: librados-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:28.412 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:28.412 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:28.449 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:28.450 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:28.450 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:28.450 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:28.450 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:28.450 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:28.639 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs2 2024-06-15T00:46:28.685 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-mirror 2024-06-15T00:46:29.231 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:43 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:29.268 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:39 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:29.319 INFO:teuthology.orchestra.run.smithi143.stdout:Package libcephfs2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:29.361 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:29.362 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:29.362 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:29.397 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: Package Arch Version Repo Size 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:29.398 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:29.399 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:29.399 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:46:29.399 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:29.399 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 2.9 M 2024-06-15T00:46:29.399 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 12 M 2024-06-15T00:46:29.399 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:29.432 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install libcephfs-devel 2024-06-15T00:46:29.938 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-mirror-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 5.4 MB/s | 2.9 MB 00:00 2024-06-15T00:46:29.939 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:29.939 INFO:teuthology.orchestra.run.smithi105.stdout:Total 5.4 MB/s | 2.9 MB 00:00 2024-06-15T00:46:29.940 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:29.949 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:29.949 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:30.012 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:44 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:30.029 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:30.030 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:30.138 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:30.138 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:30.138 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 25 k 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 25 k 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 110 k 2024-06-15T00:46:30.139 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:30.306 INFO:teuthology.orchestra.run.smithi143.stdout:libcephfs-devel-17.2.7-1008.ge47ff4b8.el9.x86_6 149 kB/s | 25 kB 00:00 2024-06-15T00:46:30.307 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:30.307 INFO:teuthology.orchestra.run.smithi143.stdout:Total 148 kB/s | 25 kB 00:00 2024-06-15T00:46:30.307 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:30.310 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:30.310 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:30.312 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:30.312 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:30.388 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:30.440 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:30.486 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:30.489 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:30.514 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:30.514 INFO:teuthology.orchestra.run.smithi105.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:46:30.514 INFO:teuthology.orchestra.run.smithi105.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-15T00:46:30.514 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-15T00:46:30.514 INFO:teuthology.orchestra.run.smithi105.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-15T00:46:30.514 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:30.832 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:31.059 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:31.059 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:31.059 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:31.060 INFO:teuthology.orchestra.run.smithi143.stdout: libcephfs-devel-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:31.060 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:31.060 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:31.271 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librados2 2024-06-15T00:46:31.606 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:31.606 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:31.606 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:31.606 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:31.606 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:31.606 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:31.850 DEBUG:teuthology.orchestra.run.smithi105:> sudo yum -y install rbd-nbd 2024-06-15T00:46:31.851 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:45 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:31.935 INFO:teuthology.orchestra.run.smithi143.stdout:Package librados2-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:31.977 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:31.978 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:31.978 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:32.052 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install librbd1 2024-06-15T00:46:32.435 INFO:teuthology.orchestra.run.smithi105.stdout:Last metadata expiration check: 0:01:42 ago on Sat 15 Jun 2024 12:44:50 AM UTC. 2024-06-15T00:46:32.564 INFO:teuthology.orchestra.run.smithi105.stdout:Dependencies resolved. 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout: Package Architecture Version Repository Size 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout:Installing: 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 165 k 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction Summary 2024-06-15T00:46:32.565 INFO:teuthology.orchestra.run.smithi105.stdout:================================================================================ 2024-06-15T00:46:32.566 INFO:teuthology.orchestra.run.smithi105.stdout:Install 1 Package 2024-06-15T00:46:32.566 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:32.566 INFO:teuthology.orchestra.run.smithi105.stdout:Total download size: 165 k 2024-06-15T00:46:32.566 INFO:teuthology.orchestra.run.smithi105.stdout:Installed size: 477 k 2024-06-15T00:46:32.566 INFO:teuthology.orchestra.run.smithi105.stdout:Downloading Packages: 2024-06-15T00:46:32.634 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:46 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:32.719 INFO:teuthology.orchestra.run.smithi143.stdout:Package librbd1-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:32.761 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:32.762 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:32.762 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:32.768 INFO:teuthology.orchestra.run.smithi105.stdout:rbd-nbd-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 818 kB/s | 165 kB 00:00 2024-06-15T00:46:32.769 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:32.769 INFO:teuthology.orchestra.run.smithi105.stdout:Total 810 kB/s | 165 kB 00:00 2024-06-15T00:46:32.769 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction check 2024-06-15T00:46:32.777 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction check succeeded. 2024-06-15T00:46:32.777 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction test 2024-06-15T00:46:32.835 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rados 2024-06-15T00:46:32.843 INFO:teuthology.orchestra.run.smithi105.stdout:Transaction test succeeded. 2024-06-15T00:46:32.843 INFO:teuthology.orchestra.run.smithi105.stdout:Running transaction 2024-06-15T00:46:32.996 INFO:teuthology.orchestra.run.smithi105.stdout: Preparing : 1/1 2024-06-15T00:46:33.090 INFO:teuthology.orchestra.run.smithi105.stdout: Installing : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:33.419 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:47 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:33.502 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rados-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:33.518 INFO:teuthology.orchestra.run.smithi105.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:33.544 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:33.545 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:33.545 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:33.626 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rgw 2024-06-15T00:46:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: Verifying : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:33.771 INFO:teuthology.orchestra.run.smithi105.stdout:Installed: 2024-06-15T00:46:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:33.771 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:46:33.772 INFO:teuthology.orchestra.run.smithi105.stdout:Complete! 2024-06-15T00:46:34.030 DEBUG:teuthology.parallel:result is None 2024-06-15T00:46:34.209 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:48 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:34.293 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rgw-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:34.334 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:34.335 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:34.336 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:34.418 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-cephfs 2024-06-15T00:46:34.994 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:48 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:35.079 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-cephfs-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:35.120 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:35.121 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:35.193 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install python3-rbd 2024-06-15T00:46:35.771 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:49 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:35.856 INFO:teuthology.orchestra.run.smithi143.stdout:Package python3-rbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 is already installed. 2024-06-15T00:46:35.898 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:35.899 INFO:teuthology.orchestra.run.smithi143.stdout:Nothing to do. 2024-06-15T00:46:35.899 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:35.975 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-fuse 2024-06-15T00:46:36.552 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:50 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:36.680 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:36.680 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 82 k 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 82 k 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 222 k 2024-06-15T00:46:36.681 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:36.885 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-fuse-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 404 kB/s | 82 kB 00:00 2024-06-15T00:46:36.886 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:36.886 INFO:teuthology.orchestra.run.smithi143.stdout:Total 401 kB/s | 82 kB 00:00 2024-06-15T00:46:36.886 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:36.892 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:36.892 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:36.955 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:36.955 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:37.100 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:37.190 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:37.565 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:37.818 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:37.818 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:37.818 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:37.818 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-fuse-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:37.818 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:37.818 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:38.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-mirror 2024-06-15T00:46:38.613 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:52 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:38.741 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:38.741 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:38.741 INFO:teuthology.orchestra.run.smithi143.stdout: Package Arch Version Repo Size 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 2.9 M 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:46:38.742 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:38.743 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 2.9 M 2024-06-15T00:46:38.743 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 12 M 2024-06-15T00:46:38.743 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:39.114 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-mirror-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 7.9 MB/s | 2.9 MB 00:00 2024-06-15T00:46:39.115 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:39.115 INFO:teuthology.orchestra.run.smithi143.stdout:Total 7.9 MB/s | 2.9 MB 00:00 2024-06-15T00:46:39.115 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:39.124 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:39.125 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:39.205 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:39.205 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:39.637 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:39.698 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:39.722 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:39.722 INFO:teuthology.orchestra.run.smithi143.stdout:Glob pattern passed to enable, but globs are not supported for this. 2024-06-15T00:46:39.722 INFO:teuthology.orchestra.run.smithi143.stdout:Invalid unit name "ceph-rbd-mirror@*.service" escaped as "ceph-rbd-mirror@\x2a.service". 2024-06-15T00:46:39.722 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-15T00:46:39.723 INFO:teuthology.orchestra.run.smithi143.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-rbd-mirror.target → /usr/lib/systemd/system/ceph-rbd-mirror.target. 2024-06-15T00:46:39.723 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:40.805 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:40.805 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:40.805 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:40.805 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-mirror-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:40.805 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:40.805 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:41.027 DEBUG:teuthology.orchestra.run.smithi143:> sudo yum -y install rbd-nbd 2024-06-15T00:46:41.621 INFO:teuthology.orchestra.run.smithi143.stdout:Last metadata expiration check: 0:01:55 ago on Sat 15 Jun 2024 12:44:46 AM UTC. 2024-06-15T00:46:41.746 INFO:teuthology.orchestra.run.smithi143.stdout:Dependencies resolved. 2024-06-15T00:46:41.747 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:41.747 INFO:teuthology.orchestra.run.smithi143.stdout: Package Architecture Version Repository Size 2024-06-15T00:46:41.747 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:41.747 INFO:teuthology.orchestra.run.smithi143.stdout:Installing: 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd x86_64 2:17.2.7-1008.ge47ff4b8.el9 ceph 165 k 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction Summary 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout:================================================================================ 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout:Install 1 Package 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout:Total download size: 165 k 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout:Installed size: 477 k 2024-06-15T00:46:41.748 INFO:teuthology.orchestra.run.smithi143.stdout:Downloading Packages: 2024-06-15T00:46:41.968 INFO:teuthology.orchestra.run.smithi143.stdout:rbd-nbd-17.2.7-1008.ge47ff4b8.el9.x86_64.rpm 750 kB/s | 165 kB 00:00 2024-06-15T00:46:41.969 INFO:teuthology.orchestra.run.smithi143.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:46:41.970 INFO:teuthology.orchestra.run.smithi143.stdout:Total 745 kB/s | 165 kB 00:00 2024-06-15T00:46:41.970 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction check 2024-06-15T00:46:41.977 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction check succeeded. 2024-06-15T00:46:41.977 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction test 2024-06-15T00:46:42.040 INFO:teuthology.orchestra.run.smithi143.stdout:Transaction test succeeded. 2024-06-15T00:46:42.040 INFO:teuthology.orchestra.run.smithi143.stdout:Running transaction 2024-06-15T00:46:42.199 INFO:teuthology.orchestra.run.smithi143.stdout: Preparing : 1/1 2024-06-15T00:46:42.308 INFO:teuthology.orchestra.run.smithi143.stdout: Installing : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:42.748 INFO:teuthology.orchestra.run.smithi143.stdout: Running scriptlet: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:43.002 INFO:teuthology.orchestra.run.smithi143.stdout: Verifying : rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 1/1 2024-06-15T00:46:43.003 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:43.003 INFO:teuthology.orchestra.run.smithi143.stdout:Installed: 2024-06-15T00:46:43.003 INFO:teuthology.orchestra.run.smithi143.stdout: rbd-nbd-2:17.2.7-1008.ge47ff4b8.el9.x86_64 2024-06-15T00:46:43.003 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:46:43.003 INFO:teuthology.orchestra.run.smithi143.stdout:Complete! 2024-06-15T00:46:43.333 DEBUG:teuthology.parallel:result is None 2024-06-15T00:46:43.334 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:46:43.334 INFO:teuthology.packaging:ref: None 2024-06-15T00:46:43.334 INFO:teuthology.packaging:tag: None 2024-06-15T00:46:43.334 INFO:teuthology.packaging:branch: quincy 2024-06-15T00:46:43.334 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:46:43.334 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-15T00:46:43.576 DEBUG:teuthology.orchestra.run.smithi105:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-15T00:46:43.602 INFO:teuthology.orchestra.run.smithi105.stdout:17.2.7-1008.ge47ff4b8.el9 2024-06-15T00:46:43.603 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1008.ge47ff4b8.el9 2024-06-15T00:46:43.603 INFO:teuthology.task.install:The correct ceph version 17.2.7-1008.ge47ff4b8 is installed. 2024-06-15T00:46:43.605 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-06-15T00:46:43.605 INFO:teuthology.packaging:ref: None 2024-06-15T00:46:43.605 INFO:teuthology.packaging:tag: None 2024-06-15T00:46:43.605 INFO:teuthology.packaging:branch: quincy 2024-06-15T00:46:43.605 INFO:teuthology.packaging:sha1: 6f4b09560ed399c3f139402a02f389c2fc8bb2cc 2024-06-15T00:46:43.605 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F9%2Fx86_64&ref=quincy 2024-06-15T00:46:43.843 DEBUG:teuthology.orchestra.run.smithi143:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-06-15T00:46:43.865 INFO:teuthology.orchestra.run.smithi143.stdout:17.2.7-1008.ge47ff4b8.el9 2024-06-15T00:46:43.865 INFO:teuthology.packaging:The installed version of ceph is 17.2.7-1008.ge47ff4b8.el9 2024-06-15T00:46:43.865 INFO:teuthology.task.install:The correct ceph version 17.2.7-1008.ge47ff4b8 is installed. 2024-06-15T00:46:43.867 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-06-15T00:46:43.867 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:46:43.868 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-15T00:46:43.897 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:46:43.897 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-06-15T00:46:43.930 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-06-15T00:46:43.931 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:46:43.931 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/daemon-helper 2024-06-15T00:46:43.964 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-15T00:46:44.030 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:46:44.031 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/daemon-helper 2024-06-15T00:46:44.058 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-06-15T00:46:44.122 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-06-15T00:46:44.122 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:46:44.122 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-15T00:46:44.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-15T00:46:44.214 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:46:44.214 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/adjust-ulimits 2024-06-15T00:46:44.239 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-06-15T00:46:44.306 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-06-15T00:46:44.307 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:46:44.307 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/usr/bin/stdin-killer 2024-06-15T00:46:44.332 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-15T00:46:44.398 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:46:44.398 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/usr/bin/stdin-killer 2024-06-15T00:46:44.422 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-06-15T00:46:44.486 INFO:teuthology.run_tasks:Running task print... 2024-06-15T00:46:44.497 INFO:teuthology.task.print:**** done install task... 2024-06-15T00:46:44.497 INFO:teuthology.run_tasks:Running task cephadm... 2024-06-15T00:46:44.632 INFO:tasks.cephadm:Config: {'cephadm_branch': 'quincy', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'conf': {'osd': {'osd_class_default_list': '*', 'osd_class_load_list': '*', 'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}, 'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}}, 'image': 'quay.ceph.io/ceph-ci/ceph:quincy', 'roleless': True, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'FS_DEGRADED', 'FS_INLINE_DATA_DEPRECATED', 'FS_WITH_FAILED_MDS', 'MDS_ALL_DOWN', 'MDS_DAMAGE', 'MDS_DEGRADED', 'MDS_FAILED', 'MDS_INSUFFICIENT_STANDBY', 'MDS_UP_LESS_THAN_MAX', 'POOL_APP_NOT_ENABLED', 'overall HEALTH_', 'Replacing daemon', 'OSD_DOWN', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '6f4b09560ed399c3f139402a02f389c2fc8bb2cc'} 2024-06-15T00:46:44.632 INFO:tasks.cephadm:Cluster image is quay.ceph.io/ceph-ci/ceph:quincy 2024-06-15T00:46:44.632 INFO:tasks.cephadm:Cluster fsid is bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:46:44.632 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-06-15T00:46:44.633 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-06-15T00:46:44.633 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi105': '172.21.15.105', 'mon.smithi143': '172.21.15.143'} 2024-06-15T00:46:44.633 INFO:tasks.cephadm:Normalizing hostnames... 2024-06-15T00:46:44.633 DEBUG:teuthology.orchestra.run.smithi105:> sudo hostname $(hostname -s) 2024-06-15T00:46:44.659 DEBUG:teuthology.orchestra.run.smithi143:> sudo hostname $(hostname -s) 2024-06-15T00:46:44.684 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref quincy)... 2024-06-15T00:46:44.685 DEBUG:teuthology.orchestra.run.smithi105:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-15T00:46:45.000 INFO:teuthology.orchestra.run.smithi105.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 15 00:46 /home/ubuntu/cephtest/cephadm 2024-06-15T00:46:45.001 DEBUG:teuthology.orchestra.run.smithi143:> curl --silent https://raw.githubusercontent.com/ceph/ceph/quincy/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-06-15T00:46:45.255 INFO:teuthology.orchestra.run.smithi143.stdout:-rw-r--r--. 1 ubuntu ubuntu 375153 Jun 15 00:46 /home/ubuntu/cephtest/cephadm 2024-06-15T00:46:45.256 DEBUG:teuthology.orchestra.run.smithi105:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-15T00:46:45.273 DEBUG:teuthology.orchestra.run.smithi143:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-06-15T00:46:45.299 INFO:tasks.cephadm:Pulling image quay.ceph.io/ceph-ci/ceph:quincy on all hosts... 2024-06-15T00:46:45.299 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-15T00:46:45.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy pull 2024-06-15T00:46:46.116 INFO:teuthology.orchestra.run.smithi105.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:46:46.230 INFO:teuthology.orchestra.run.smithi143.stderr:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:47:14.777 INFO:teuthology.orchestra.run.smithi143.stdout:{ 2024-06-15T00:47:14.777 INFO:teuthology.orchestra.run.smithi143.stdout: "ceph_version": "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)", 2024-06-15T00:47:14.777 INFO:teuthology.orchestra.run.smithi143.stdout: "image_id": "37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95", 2024-06-15T00:47:14.777 INFO:teuthology.orchestra.run.smithi143.stdout: "repo_digests": [ 2024-06-15T00:47:14.777 INFO:teuthology.orchestra.run.smithi143.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580" 2024-06-15T00:47:14.777 INFO:teuthology.orchestra.run.smithi143.stdout: ] 2024-06-15T00:47:14.778 INFO:teuthology.orchestra.run.smithi143.stdout:} 2024-06-15T00:47:16.947 INFO:teuthology.orchestra.run.smithi105.stdout:{ 2024-06-15T00:47:16.948 INFO:teuthology.orchestra.run.smithi105.stdout: "ceph_version": "ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable)", 2024-06-15T00:47:16.948 INFO:teuthology.orchestra.run.smithi105.stdout: "image_id": "37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95", 2024-06-15T00:47:16.948 INFO:teuthology.orchestra.run.smithi105.stdout: "repo_digests": [ 2024-06-15T00:47:16.948 INFO:teuthology.orchestra.run.smithi105.stdout: "quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580" 2024-06-15T00:47:16.948 INFO:teuthology.orchestra.run.smithi105.stdout: ] 2024-06-15T00:47:16.948 INFO:teuthology.orchestra.run.smithi105.stdout:} 2024-06-15T00:47:16.966 DEBUG:teuthology.orchestra.run.smithi105:> sudo mkdir -p /etc/ceph 2024-06-15T00:47:17.012 DEBUG:teuthology.orchestra.run.smithi143:> sudo mkdir -p /etc/ceph 2024-06-15T00:47:17.045 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 777 /etc/ceph 2024-06-15T00:47:17.069 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 777 /etc/ceph 2024-06-15T00:47:17.114 INFO:tasks.cephadm:Writing seed config... 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] osd_class_default_list = * 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] osd_class_load_list = * 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-06-15T00:47:17.115 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [client] debug client = 20 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-06-15T00:47:17.116 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-06-15T00:47:17.117 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-06-15T00:47:17.118 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-06-15T00:47:17.118 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-06-15T00:47:17.118 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-06-15T00:47:17.118 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:47:17.119 DEBUG:teuthology.orchestra.run.smithi105:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-06-15T00:47:17.134 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false log to journald = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = bd0fff6c-2ab0-11ef-bc9d-c7b262605968 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd mclock profile = high_recovery_ops osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true osd_class_default_list = * osd_class_load_list = * bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-06-15T00:47:17.134 DEBUG:teuthology.orchestra.run.smithi105:mon.smithi105> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service 2024-06-15T00:47:17.177 INFO:tasks.cephadm:Bootstrapping... 2024-06-15T00:47:17.177 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy -v bootstrap --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.105 --skip-admin-label && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:47:17.377 INFO:teuthology.orchestra.run.smithi105.stdout:-------------------------------------------------------------------------------- 2024-06-15T00:47:17.378 INFO:teuthology.orchestra.run.smithi105.stdout:cephadm ['--image', 'quay.ceph.io/ceph-ci/ceph:quincy', '-v', 'bootstrap', '--fsid', 'bd0fff6c-2ab0-11ef-bc9d-c7b262605968', '--config', '/home/ubuntu/cephtest/seed.ceph.conf', '--output-config', '/etc/ceph/ceph.conf', '--output-keyring', '/etc/ceph/ceph.client.admin.keyring', '--output-pub-ssh-key', '/home/ubuntu/cephtest/ceph.pub', '--mon-ip', '172.21.15.105', '--skip-admin-label'] 2024-06-15T00:47:17.402 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 5.1.0 2024-06-15T00:47:17.403 INFO:teuthology.orchestra.run.smithi105.stderr:Specifying an fsid for your cluster offers no advantages and may increase the likelihood of fsid conflicts. 2024-06-15T00:47:17.403 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying podman|docker is present... 2024-06-15T00:47:17.426 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 5.1.0 2024-06-15T00:47:17.426 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying lvm2 is present... 2024-06-15T00:47:17.426 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying time synchronization is in place... 2024-06-15T00:47:17.434 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-15T00:47:17.434 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-15T00:47:17.441 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-15T00:47:17.441 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-06-15T00:47:17.448 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout enabled 2024-06-15T00:47:17.455 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout active 2024-06-15T00:47:17.455 INFO:teuthology.orchestra.run.smithi105.stdout:Unit chronyd.service is enabled and running 2024-06-15T00:47:17.455 INFO:teuthology.orchestra.run.smithi105.stdout:Repeating the final host check... 2024-06-15T00:47:17.479 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 5.1.0 2024-06-15T00:47:17.479 INFO:teuthology.orchestra.run.smithi105.stdout:podman (/bin/podman) version 5.1.0 is present 2024-06-15T00:47:17.479 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl is present 2024-06-15T00:47:17.479 INFO:teuthology.orchestra.run.smithi105.stdout:lvcreate is present 2024-06-15T00:47:17.486 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled chrony.service 2024-06-15T00:47:17.486 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to get unit file state for chrony.service: No such file or directory 2024-06-15T00:47:17.493 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active chrony.service 2024-06-15T00:47:17.493 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-06-15T00:47:17.499 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout enabled 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout active 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:Unit chronyd.service is enabled and running 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:Host looks OK 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:Cluster fsid: bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:Acquiring lock 139903349420480 on /run/cephadm/bd0fff6c-2ab0-11ef-bc9d-c7b262605968.lock 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:Lock 139903349420480 acquired on /run/cephadm/bd0fff6c-2ab0-11ef-bc9d-c7b262605968.lock 2024-06-15T00:47:17.507 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying IP 172.21.15.105 port 3300 ... 2024-06-15T00:47:17.508 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying IP 172.21.15.105 port 6789 ... 2024-06-15T00:47:17.508 INFO:teuthology.orchestra.run.smithi105.stdout:Base mon IP(s) is [172.21.15.105:3300, 172.21.15.105:6789], mon addrv is [v2:172.21.15.105:3300,v1:172.21.15.105:6789] 2024-06-15T00:47:17.511 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.105 metric 100 2024-06-15T00:47:17.511 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.105 metric 100 2024-06-15T00:47:17.514 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout ::1 dev lo proto kernel metric 256 pref medium 2024-06-15T00:47:17.514 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-06-15T00:47:17.514 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1798sec hoplimit 64 pref medium 2024-06-15T00:47:17.517 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-06-15T00:47:17.517 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout inet6 ::1/128 scope host 2024-06-15T00:47:17.517 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-15T00:47:17.517 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-06-15T00:47:17.517 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout inet6 fe80::ec4:7aff:fe8f:d429/64 scope link 2024-06-15T00:47:17.517 INFO:teuthology.orchestra.run.smithi105.stdout:/sbin/ip: stdout valid_lft forever preferred_lft forever 2024-06-15T00:47:17.518 INFO:teuthology.orchestra.run.smithi105.stdout:Mon IP `172.21.15.105` is in CIDR network `172.21.0.0/20` 2024-06-15T00:47:17.518 INFO:teuthology.orchestra.run.smithi105.stdout:Mon IP `172.21.15.105` is in CIDR network `172.21.0.0/20` 2024-06-15T00:47:17.518 INFO:teuthology.orchestra.run.smithi105.stdout:Inferred mon public CIDR from local network configuration ['172.21.0.0/20', '172.21.0.0/20'] 2024-06-15T00:47:17.518 INFO:teuthology.orchestra.run.smithi105.stdout:Internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-06-15T00:47:17.519 INFO:teuthology.orchestra.run.smithi105.stdout:Pulling container image quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:47:17.989 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stdout 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 2024-06-15T00:47:17.989 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Trying to pull quay.ceph.io/ceph-ci/ceph:quincy... 2024-06-15T00:47:17.989 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Getting image source signatures 2024-06-15T00:47:17.989 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying blob sha256:ae1bd4835d962ece7feffb132e988e26ca074ae51fc4b41b3d8d10a6caf0861f 2024-06-15T00:47:17.990 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying blob sha256:9065cd298533077f0006921bb12bd331d24f6c140cc8fa972c2e6c4e382038d1 2024-06-15T00:47:17.990 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Copying config sha256:37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 2024-06-15T00:47:17.990 INFO:teuthology.orchestra.run.smithi105.stdout:/bin/podman: stderr Writing manifest to image destination 2024-06-15T00:47:18.786 INFO:teuthology.orchestra.run.smithi105.stdout:ceph: stdout ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable) 2024-06-15T00:47:18.786 INFO:teuthology.orchestra.run.smithi105.stdout:Ceph version: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable) 2024-06-15T00:47:18.786 INFO:teuthology.orchestra.run.smithi105.stdout:Extracting ceph user uid/gid from container image... 2024-06-15T00:47:19.708 INFO:teuthology.orchestra.run.smithi105.stdout:stat: stdout 167 167 2024-06-15T00:47:19.708 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial keys... 2024-06-15T00:47:20.564 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQCX5GxmaZv7OhAAOqxYiElQRVL+lm//2pZVcg== 2024-06-15T00:47:21.535 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQCY5GxmiCVcMhAAQB+HJf2I8inCfOFwfnwzbA== 2024-06-15T00:47:22.383 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph-authtool: stdout AQCZ5Gxm+Qm0MhAAJGSXIP3lNwy2MDbL1lbVLg== 2024-06-15T00:47:22.384 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial monmap... 2024-06-15T00:47:23.314 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-15T00:47:23.314 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout setting min_mon_release = octopus 2024-06-15T00:47:23.314 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: set fsid to bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: stdout /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout:monmaptool for smithi105 [v2:172.21.15.105:3300,v1:172.21.15.105:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout:setting min_mon_release = octopus 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: set fsid to bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:47:23.315 INFO:teuthology.orchestra.run.smithi105.stdout:Creating mon... 2024-06-15T00:47:24.287 INFO:teuthology.orchestra.run.smithi105.stdout:create mon.smithi105 on 2024-06-15T00:47:24.486 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Removed "/etc/systemd/system/multi-user.target.wants/ceph.target". 2024-06-15T00:47:24.628 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-06-15T00:47:24.774 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/multi-user.target.wants/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968.target. 2024-06-15T00:47:24.774 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph.target.wants/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968.target → /etc/systemd/system/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968.target. 2024-06-15T00:47:24.936 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105 2024-06-15T00:47:24.936 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service: Unit ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service not loaded. 2024-06-15T00:47:25.084 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968.target.wants/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service → /etc/systemd/system/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@.service. 2024-06-15T00:47:25.629 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:25 smithi105 ceph-mon[28509]: mon.smithi105 is new leader, mons smithi105 in quorum (ranks 0) 2024-06-15T00:47:25.630 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:25 smithi105 bash[28482]: 01bebab2e2e527e82a36ba17a55c4efa6cc808e2b4f2798e7ea74d291acdcd46 2024-06-15T00:47:25.644 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:47:25.644 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-06-15T00:47:25.653 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:47:25.653 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-06-15T00:47:25.653 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-06-15T00:47:25.653 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to enable service . firewalld.service is not available 2024-06-15T00:47:25.653 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mon to start... 2024-06-15T00:47:25.654 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mon... 2024-06-15T00:47:25.898 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:25 smithi105 systemd[1]: Started Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968. 2024-06-15T00:47:26.629 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:26 smithi105 ceph-mon[28509]: mon.smithi105 is new leader, mons smithi105 in quorum (ranks 0) 2024-06-15T00:47:26.630 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:26 smithi105 ceph-mon[28509]: monmap e1: 1 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:47:26.630 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:26 smithi105 ceph-mon[28509]: fsmap 2024-06-15T00:47:26.630 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:26 smithi105 ceph-mon[28509]: osdmap e1: 0 total, 0 up, 0 in 2024-06-15T00:47:26.630 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:26 smithi105 ceph-mon[28509]: mgrmap e1: no daemons active 2024-06-15T00:47:26.630 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:26 smithi105 ceph-mon[28509]: from='client.? 172.21.15.105:0/4211395781' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-06-15T00:47:26.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout cluster: 2024-06-15T00:47:26.744 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout id: bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:26.744 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout health: HEALTH_OK 2024-06-15T00:47:26.744 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:26.744 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout services: 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon: 1 daemons, quorum smithi105 (age 0.565167s) 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr: no daemons active 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd: 0 osds: 0 up, 0 in 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout data: 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout pools: 0 pools, 0 pgs 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout objects: 0 objects, 0 B 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout usage: 0 B used, 0 B / 0 B avail 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout pgs: 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.057+0000 7f33e0f66640 1 Processor -- start 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33e0f66640 1 -- start start 2024-06-15T00:47:26.745 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33e0f66640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc1046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33e0f66640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc104c00 con 0x7f33dc075c40 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33da59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc1046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33da59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc1046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60388/0 (socket says 172.21.15.105:60388) 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33da59c640 1 -- 172.21.15.105:0/1530981090 learned_addr learned my addr 172.21.15.105:0/1530981090 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.058+0000 7f33da59c640 1 -- 172.21.15.105:0/1530981090 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc104d40 con 0x7f33dc075c40 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.061+0000 7f33da59c640 1 --2- 172.21.15.105:0/1530981090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc1046c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009840 tx=0x7f33cc0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=82df0745f5333e05 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.061+0000 7f33d9d9b640 1 -- 172.21.15.105:0/1530981090 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc032440 con 0x7f33dc075c40 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.061+0000 7f33d9d9b640 1 -- 172.21.15.105:0/1530981090 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc0325a0 con 0x7f33dc075c40 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.061+0000 7f33d9d9b640 1 -- 172.21.15.105:0/1530981090 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc0328d0 con 0x7f33dc075c40 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.061+0000 7f33e0f66640 1 -- 172.21.15.105:0/1530981090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 msgr2=0x7f33dc1046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.061+0000 7f33e0f66640 1 --2- 172.21.15.105:0/1530981090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc1046c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f33cc009840 tx=0x7f33cc0091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 -- 172.21.15.105:0/1530981090 shutdown_connections 2024-06-15T00:47:26.746 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 --2- 172.21.15.105:0/1530981090 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc1046c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 -- 172.21.15.105:0/1530981090 >> 172.21.15.105:0/1530981090 conn(0x7f33dc0fc190 msgr2=0x7f33dc0fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 -- 172.21.15.105:0/1530981090 shutdown_connections 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 -- 172.21.15.105:0/1530981090 wait complete. 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 Processor -- start 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 -- start start 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc110090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.062+0000 7f33e0f66640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc1105d0 con 0x7f33dc075c40 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33da59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc110090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33da59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc110090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60398/0 (socket says 172.21.15.105:60398) 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33da59c640 1 -- 172.21.15.105:0/4211395781 learned_addr learned my addr 172.21.15.105:0/4211395781 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33da59c640 1 -- 172.21.15.105:0/4211395781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc009450 con 0x7f33dc075c40 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33da59c640 1 --2- 172.21.15.105:0/4211395781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc110090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f33cc02a700 tx=0x7f33cc032000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:26.747 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc031be0 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33cc031d40 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc1107d0 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc03b5a0 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.063+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc110be0 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.064+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f33cc039040 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.064+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f33cc041030 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.064+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc04b480 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.068+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f33cc037070 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.130+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f33dc1098a0 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.131+0000 7f33d2ffd640 1 -- 172.21.15.105:0/4211395781 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+325 (secure 0 0 0) 0x7f33cc03bbd0 con 0x7f33dc075c40 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 msgr2=0x7f33dc110090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 --2- 172.21.15.105:0/4211395781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc110090 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f33cc02a700 tx=0x7f33cc032000 comp rx=0 tx=0).stop 2024-06-15T00:47:26.748 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 shutdown_connections 2024-06-15T00:47:26.749 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 --2- 172.21.15.105:0/4211395781 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f33dc075c40 0x7f33dc110090 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:26.749 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 >> 172.21.15.105:0/4211395781 conn(0x7f33dc0fc190 msgr2=0x7f33dc0fc560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:26.749 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 shutdown_connections 2024-06-15T00:47:26.749 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:26.132+0000 7f33e0f66640 1 -- 172.21.15.105:0/4211395781 wait complete. 2024-06-15T00:47:26.749 INFO:teuthology.orchestra.run.smithi105.stdout:mon is available 2024-06-15T00:47:26.749 INFO:teuthology.orchestra.run.smithi105.stdout:Assimilating anything we can from ceph.conf... 2024-06-15T00:47:27.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:27.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [global] 2024-06-15T00:47:27.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout fsid = bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:27.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_host = [v2:172.21.15.105:3300,v1:172.21.15.105:6789] 2024-06-15T00:47:27.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-15T00:47:27.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [osd] 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f0842995640 1 Processor -- start 2024-06-15T00:47:27.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f0842995640 1 -- start start 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f0842995640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f0842995640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c1012e0 con 0x7f083c100990 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f083bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f083bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60406/0 (socket says 172.21.15.105:60406) 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f083bfff640 1 -- 172.21.15.105:0/295424488 learned_addr learned my addr 172.21.15.105:0/295424488 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.162+0000 7f083bfff640 1 -- 172.21.15.105:0/295424488 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f083c101420 con 0x7f083c100990 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.163+0000 7f083bfff640 1 --2- 172.21.15.105:0/295424488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c100da0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f082c00baf0 tx=0x7f082c00b460 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a1f95d626ce0d7a9 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.163+0000 7f083affd640 1 -- 172.21.15.105:0/295424488 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f082c0327a0 con 0x7f083c100990 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.163+0000 7f083affd640 1 -- 172.21.15.105:0/295424488 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c032900 con 0x7f083c100990 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.163+0000 7f083affd640 1 -- 172.21.15.105:0/295424488 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f082c032c30 con 0x7f083c100990 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 -- 172.21.15.105:0/295424488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 msgr2=0x7f083c100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:27.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 --2- 172.21.15.105:0/295424488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c100da0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 crypto rx=0x7f082c00baf0 tx=0x7f082c00b460 comp rx=0 tx=0).stop 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 -- 172.21.15.105:0/295424488 shutdown_connections 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 --2- 172.21.15.105:0/295424488 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c100da0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 -- 172.21.15.105:0/295424488 >> 172.21.15.105:0/295424488 conn(0x7f083c0fc150 msgr2=0x7f083c0fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 -- 172.21.15.105:0/295424488 shutdown_connections 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.164+0000 7f0842995640 1 -- 172.21.15.105:0/295424488 wait complete. 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f0842995640 1 Processor -- start 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f0842995640 1 -- start start 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f0842995640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c10e140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f0842995640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f083c10e680 con 0x7f083c100990 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f083bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c10e140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f083bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c10e140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60414/0 (socket says 172.21.15.105:60414) 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f083bfff640 1 -- 172.21.15.105:0/4044762923 learned_addr learned my addr 172.21.15.105:0/4044762923 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f083bfff640 1 -- 172.21.15.105:0/4044762923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082c00b700 con 0x7f083c100990 2024-06-15T00:47:27.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.165+0000 7f083bfff640 1 --2- 172.21.15.105:0/4044762923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c10e140 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f082c00baf0 tx=0x7f082c03a000 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f082c03c720 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f082c03c880 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f082c03cb70 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f083c10e880 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f083c10ecf0 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f082c045070 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.166+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f082c051a50 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.167+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f083c04b480 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.169+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f083c10ecf0 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.227+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f083c101cd0 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.231+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 7 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f082c03cd70 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.231+0000 7f0838ff9640 1 -- 172.21.15.105:0/4044762923 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+431 (secure 0 0 0) 0x7f082c04c070 con 0x7f083c100990 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 msgr2=0x7f083c10e140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 --2- 172.21.15.105:0/4044762923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c10e140 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7f082c00baf0 tx=0x7f082c03a000 comp rx=0 tx=0).stop 2024-06-15T00:47:27.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 shutdown_connections 2024-06-15T00:47:27.797 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 --2- 172.21.15.105:0/4044762923 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f083c100990 0x7f083c10e140 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:27.797 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 >> 172.21.15.105:0/4044762923 conn(0x7f083c0fc150 msgr2=0x7f083c104e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:27.797 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 shutdown_connections 2024-06-15T00:47:27.797 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:27.232+0000 7f0842995640 1 -- 172.21.15.105:0/4044762923 wait complete. 2024-06-15T00:47:27.797 INFO:teuthology.orchestra.run.smithi105.stdout:Generating new minimal ceph.conf... 2024-06-15T00:47:27.901 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:27 smithi105 ceph-mon[28509]: from='client.? 172.21.15.105:0/4044762923' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-15T00:47:27.901 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:27 smithi105 ceph-mon[28509]: from='client.? 172.21.15.105:0/4044762923' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-06-15T00:47:28.696 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed6d960640 1 Processor -- start 2024-06-15T00:47:28.696 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed6d960640 1 -- start start 2024-06-15T00:47:28.696 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed6d960640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:28.696 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed6d960640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed68102440 con 0x7fed68101af0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed66ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed66ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60416/0 (socket says 172.21.15.105:60416) 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.169+0000 7fed66ffd640 1 -- 172.21.15.105:0/1938878766 learned_addr learned my addr 172.21.15.105:0/1938878766 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.170+0000 7fed66ffd640 1 -- 172.21.15.105:0/1938878766 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed68102580 con 0x7fed68101af0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.170+0000 7fed66ffd640 1 --2- 172.21.15.105:0/1938878766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68101f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed58009840 tx=0x7fed580091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c075697f36efd19d server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed667fc640 1 -- 172.21.15.105:0/1938878766 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed58032440 con 0x7fed68101af0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed667fc640 1 -- 172.21.15.105:0/1938878766 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fed58032a10 con 0x7fed68101af0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed667fc640 1 -- 172.21.15.105:0/1938878766 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed58031a00 con 0x7fed68101af0 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 -- 172.21.15.105:0/1938878766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 msgr2=0x7fed68101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 --2- 172.21.15.105:0/1938878766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68101f00 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fed58009840 tx=0x7fed580091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 -- 172.21.15.105:0/1938878766 shutdown_connections 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 --2- 172.21.15.105:0/1938878766 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68101f00 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:28.697 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 -- 172.21.15.105:0/1938878766 >> 172.21.15.105:0/1938878766 conn(0x7fed680fd330 msgr2=0x7fed680ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 -- 172.21.15.105:0/1938878766 shutdown_connections 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.171+0000 7fed6d960640 1 -- 172.21.15.105:0/1938878766 wait complete. 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed6d960640 1 Processor -- start 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed6d960640 1 -- start start 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed6d960640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed6d960640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed68110dc0 con 0x7fed68101af0 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed66ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed66ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68110880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60422/0 (socket says 172.21.15.105:60422) 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed66ffd640 1 -- 172.21.15.105:0/1379398054 learned_addr learned my addr 172.21.15.105:0/1379398054 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.172+0000 7fed66ffd640 1 -- 172.21.15.105:0/1379398054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58009450 con 0x7fed68101af0 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed66ffd640 1 --2- 172.21.15.105:0/1379398054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68110880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed580313e0 tx=0x7fed58008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed5803d450 con 0x7fed68101af0 2024-06-15T00:47:28.698 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed68110fc0 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fed5803da20 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed5803b8d0 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed68111430 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fed58039070 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.173+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fed58046750 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.174+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed6804b480 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.176+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fed68111430 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.235+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fed68103850 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed5f7fe640 1 -- 172.21.15.105:0/1379398054 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7fed58041020 con 0x7fed68101af0 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 msgr2=0x7fed68110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 --2- 172.21.15.105:0/1379398054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68110880 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fed580313e0 tx=0x7fed58008040 comp rx=0 tx=0).stop 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 shutdown_connections 2024-06-15T00:47:28.699 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 --2- 172.21.15.105:0/1379398054 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed68101af0 0x7fed68110880 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:28.700 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 >> 172.21.15.105:0/1379398054 conn(0x7fed680fd330 msgr2=0x7fed680fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:28.700 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 shutdown_connections 2024-06-15T00:47:28.700 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:28.236+0000 7fed6d960640 1 -- 172.21.15.105:0/1379398054 wait complete. 2024-06-15T00:47:28.700 INFO:teuthology.orchestra.run.smithi105.stdout:Restarting the monitor... 2024-06-15T00:47:28.953 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:28 smithi105 ceph-mon[28509]: from='client.? 172.21.15.105:0/1379398054' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:47:28.954 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:28 smithi105 systemd[1]: Stopping Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:47:29.259 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:28 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105[28498]: 2024-06-15T00:47:28.951+0000 7fcf5154c640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi105 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:47:29.260 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:28 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105[28498]: 2024-06-15T00:47:28.951+0000 7fcf5154c640 -1 mon.smithi105@0(leader) e1 *** Got Signal Terminated *** 2024-06-15T00:47:29.260 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 podman[28753]: 2024-06-15 00:47:29.00391246 +0000 UTC m=+0.146217231 container died 01bebab2e2e527e82a36ba17a55c4efa6cc808e2b4f2798e7ea74d291acdcd46 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, org.label-schema.build-date=20240612, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, ceph=True, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-15T00:47:29.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 podman[28753]: 2024-06-15 00:47:29.360167507 +0000 UTC m=+0.502472271 container cleanup 01bebab2e2e527e82a36ba17a55c4efa6cc808e2b4f2798e7ea74d291acdcd46 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.vendor=CentOS, ceph=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612) 2024-06-15T00:47:29.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 bash[28753]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105 2024-06-15T00:47:29.977 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 podman[28765]: 2024-06-15 00:47:29.636065806 +0000 UTC m=+0.632471027 container remove 01bebab2e2e527e82a36ba17a55c4efa6cc808e2b4f2798e7ea74d291acdcd46 (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612) 2024-06-15T00:47:29.977 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 systemd[1]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service: Deactivated successfully. 2024-06-15T00:47:29.978 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 systemd[1]: Stopped Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968. 2024-06-15T00:47:29.978 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 systemd[1]: Starting Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:47:30.209 INFO:teuthology.orchestra.run.smithi105.stdout:Setting public_network to 172.21.0.0/20 in mon config section 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 podman[28858]: 2024-06-15 00:47:29.978772908 +0000 UTC m=+0.094816909 container create 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, org.label-schema.build-date=20240612) 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:29 smithi105 podman[28858]: 2024-06-15 00:47:29.901158779 +0000 UTC m=+0.017202785 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph:quincy 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 podman[28858]: 2024-06-15 00:47:30.139439651 +0000 UTC m=+0.255483656 container init 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, io.buildah.version=1.36.0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.build-date=20240612) 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 podman[28858]: 2024-06-15 00:47:30.142440279 +0000 UTC m=+0.258484278 container start 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, ceph=True, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.license=GPLv2) 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: set uid:gid to 167:167 (ceph:ceph) 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable), process ceph-mon, pid 2 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: pidfile_write: ignore empty --pid-file 2024-06-15T00:47:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: load: jerasure load: lrc 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: RocksDB version: 6.15.5 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Compile date Jun 14 2024 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: DB SUMMARY 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: DB Session ID: X8EXN935F0XJZO9E8OYQ 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: CURRENT file: CURRENT 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: IDENTITY file: IDENTITY 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: MANIFEST file: MANIFEST-000009 size: 131 Bytes 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi105/store.db dir, Total Num: 1, files: 000008.sst 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi105/store.db: 000010.log size: 96812 ; 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.error_if_exists: 0 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.create_if_missing: 0 2024-06-15T00:47:30.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.paranoid_checks: 1 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.env: 0x558687dce440 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.fs: Posix File System 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.info_log: 0x55868907f140 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_file_opening_threads: 16 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.statistics: (nil) 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.use_fsync: 0 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_log_file_size: 0 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.keep_log_file_num: 1000 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.recycle_log_file_num: 0 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.allow_fallocate: 1 2024-06-15T00:47:30.279 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.allow_mmap_reads: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.allow_mmap_writes: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.use_direct_reads: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.create_missing_column_families: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.db_log_dir: 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi105/store.db 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.advise_random_on_open: 1 2024-06-15T00:47:30.280 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.db_write_buffer_size: 0 2024-06-15T00:47:30.281 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.write_buffer_manager: 0x55868913c7e0 2024-06-15T00:47:30.281 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-15T00:47:30.281 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-15T00:47:30.281 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-15T00:47:30.281 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-15T00:47:30.282 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.rate_limiter: (nil) 2024-06-15T00:47:30.282 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-15T00:47:30.282 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.wal_recovery_mode: 2 2024-06-15T00:47:30.282 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.enable_thread_tracking: 0 2024-06-15T00:47:30.282 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.enable_pipelined_write: 0 2024-06-15T00:47:30.283 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.unordered_write: 0 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.row_cache: None 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.wal_filter: None 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.allow_ingest_behind: 0 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.preserve_deletes: 0 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.two_write_queues: 0 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.manual_wal_flush: 0 2024-06-15T00:47:30.284 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.atomic_flush: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.log_readahead_size: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.best_efforts_recovery: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.allow_data_in_errors: 0 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.db_host_id: __hostname__ 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_background_jobs: 2 2024-06-15T00:47:30.285 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_background_compactions: -1 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_subcompactions: 1 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_total_wal_size: 0 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_open_files: -1 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bytes_per_sync: 0 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-15T00:47:30.286 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_readahead_size: 0 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_background_flushes: -1 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Compression algorithms supported: 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kZSTD supported: 0 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kXpressCompression supported: 0 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kBZip2Compression supported: 0 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kLZ4Compression supported: 1 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kZlibCompression supported: 1 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kLZ4HCCompression supported: 1 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: kSnappyCompression supported: 1 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi105/store.db/MANIFEST-000009 2024-06-15T00:47:30.287 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.merge_operator: 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_filter: None 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_filter_factory: None 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.sst_partitioner_factory: None 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x558689025a68) 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: cache_index_and_filter_blocks: 1 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: pin_top_level_index_and_filter: 1 2024-06-15T00:47:30.288 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: index_type: 0 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: data_block_index_type: 0 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: index_shortening: 1 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: hash_index_allow_collision: 1 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: checksum: 1 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: no_block_cache: 0 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache: 0x55868905b1f0 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache_name: BinnedLRUCache 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache_options: 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: capacity : 536870912 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: num_shard_bits : 4 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: strict_capacity_limit : 0 2024-06-15T00:47:30.289 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: high_pri_pool_ratio: 0.000 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_cache_compressed: (nil) 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: persistent_cache: (nil) 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_size: 4096 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_size_deviation: 10 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_restart_interval: 16 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: index_block_restart_interval: 1 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: metadata_block_size: 4096 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: partition_filters: 0 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: use_delta_encoding: 1 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: whole_key_filtering: 1 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: verify_compression: 0 2024-06-15T00:47:30.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: read_amp_bytes_per_bit: 0 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: format_version: 4 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: enable_index_compression: 1 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: block_align: 0 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.write_buffer_size: 33554432 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_write_buffer_number: 2 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression: NoCompression 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression: Disabled 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.prefix_extractor: nullptr 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.num_levels: 7 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-15T00:47:30.291 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.level: 32767 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.strategy: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-15T00:47:30.292 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compression_opts.enabled: false 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.target_file_size_base: 67108864 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-15T00:47:30.293 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.arena_block_size: 4194304 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.disable_auto_compactions: 0 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-15T00:47:30.294 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.table_properties_collectors: 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.inplace_update_support: 0 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-15T00:47:30.295 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.bloom_locality: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.max_successive_merges: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.paranoid_file_checks: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.force_consistency_checks: 1 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.report_bg_io_stats: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.ttl: 2592000 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.enable_blob_files: false 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.min_blob_size: 0 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.blob_file_size: 268435456 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-15T00:47:30.296 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi105/store.db/MANIFEST-000009 succeeded,manifest_file_number is 9, next_file_number is 11, last_sequence is 5, log_number is 5,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 5 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/version_set.cc:4082] Creating manifest 13 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718412450165255, "job": 1, "event": "recovery_started", "wal_files": [10]} 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #10 mode 2 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718412450166041, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 14, "file_size": 92181, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 90276, "index_size": 265, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 709, "raw_key_size": 13225, "raw_average_key_size": 49, "raw_value_size": 83015, "raw_average_value_size": 310, "num_data_blocks": 11, "num_entries": 267, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718412450, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "147096a2-4a86-42e5-a459-5013df668939", "db_session_id": "X8EXN935F0XJZO9E8OYQ"}} 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/version_set.cc:4082] Creating manifest 15 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718412450166369, "job": 1, "event": "recovery_finished"} 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi105/store.db/000010.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55868908e700 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: DB pointer 0x55868905e000 2024-06-15T00:47:30.297 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** DB Stats ** 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** Compaction Stats [default] ** 2024-06-15T00:47:30.298 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: L0 2/0 91.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Sum 2/0 91.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** Compaction Stats [default] ** 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-15T00:47:30.299 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative compaction: 0.00 GB write, 28.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval compaction: 0.00 GB write, 28.74 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** Compaction Stats [default] ** 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: L0 2/0 91.82 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Sum 2/0 91.82 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.300 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** Compaction Stats [default] ** 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 165.2 0.00 0.00 1 0.001 0 0 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Cumulative compaction: 0.00 GB write, 28.44 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-15T00:47:30.301 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: starting mon.smithi105 rank 0 at public addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] at bind addrs [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi105 fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???) e1 preinit fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).mds e1 new map 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).mds e1 print_map 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: e1 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: legacy client fscid: -1 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: No filesystems configured 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-15T00:47:30.302 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 bash[28858]: 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 systemd[1]: Started Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968. 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mon.smithi105 is new leader, mons smithi105 in quorum (ranks 0) 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: monmap e1: 1 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: fsmap 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: osdmap e1: 0 total, 0 up, 0 in 2024-06-15T00:47:30.303 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:30 smithi105 ceph-mon[28877]: mgrmap e1: no daemons active 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.670+0000 7f4f52074640 1 Processor -- start 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.670+0000 7f4f52074640 1 -- start start 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.670+0000 7f4f52074640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c100300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.670+0000 7f4f52074640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c100840 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.671+0000 7f4f4b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c100300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.671+0000 7f4f4b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c100300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60430/0 (socket says 172.21.15.105:60430) 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.671+0000 7f4f4b7fe640 1 -- 172.21.15.105:0/3652378105 learned_addr learned my addr 172.21.15.105:0/3652378105 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:31.262 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.671+0000 7f4f4b7fe640 1 -- 172.21.15.105:0/3652378105 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f4c100980 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f4b7fe640 1 --2- 172.21.15.105:0/3652378105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c100300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009840 tx=0x7f4f3c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce89b0ae293980a4 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f4a7fc640 1 -- 172.21.15.105:0/3652378105 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3c032440 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f4a7fc640 1 -- 172.21.15.105:0/3652378105 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4f3c032a10 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f4a7fc640 1 -- 172.21.15.105:0/3652378105 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3c031a00 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f52074640 1 -- 172.21.15.105:0/3652378105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 msgr2=0x7f4f4c100300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f52074640 1 --2- 172.21.15.105:0/3652378105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c100300 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c009840 tx=0x7f4f3c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f52074640 1 -- 172.21.15.105:0/3652378105 shutdown_connections 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f52074640 1 --2- 172.21.15.105:0/3652378105 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c100300 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.673+0000 7f4f52074640 1 -- 172.21.15.105:0/3652378105 >> 172.21.15.105:0/3652378105 conn(0x7f4f4c0fb990 msgr2=0x7f4f4c0fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f52074640 1 -- 172.21.15.105:0/3652378105 shutdown_connections 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f52074640 1 -- 172.21.15.105:0/3652378105 wait complete. 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f52074640 1 Processor -- start 2024-06-15T00:47:31.263 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f52074640 1 -- start start 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f52074640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f52074640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f4c110da0 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.674+0000 7f4f4b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c110860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f4b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c110860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60436/0 (socket says 172.21.15.105:60436) 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f4b7fe640 1 -- 172.21.15.105:0/3306682296 learned_addr learned my addr 172.21.15.105:0/3306682296 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f4b7fe640 1 -- 172.21.15.105:0/3306682296 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f3c009450 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f4b7fe640 1 --2- 172.21.15.105:0/3306682296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c110860 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0313e0 tx=0x7f4f3c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3c03d450 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f4f3c03da20 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f4c110fa0 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4f3c03b8d0 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f4c111410 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.264 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7f4f3c039070 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.675+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f4f3c046750 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.676+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f4c04b480 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.678+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7f4f4c111410 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.735+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f4f4c105a30 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.740+0000 7f4f33fff640 1 -- 172.21.15.105:0/3306682296 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f4f3c041020 con 0x7f4f4c0ffef0 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.740+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 msgr2=0x7f4f4c110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.740+0000 7f4f52074640 1 --2- 172.21.15.105:0/3306682296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c110860 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 crypto rx=0x7f4f3c0313e0 tx=0x7f4f3c008040 comp rx=0 tx=0).stop 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.740+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 shutdown_connections 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.740+0000 7f4f52074640 1 --2- 172.21.15.105:0/3306682296 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4f4c0ffef0 0x7f4f4c110860 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.740+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 >> 172.21.15.105:0/3306682296 conn(0x7f4f4c0fb990 msgr2=0x7f4f4c0fc540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.741+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 shutdown_connections 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:30.741+0000 7f4f52074640 1 -- 172.21.15.105:0/3306682296 wait complete. 2024-06-15T00:47:31.265 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote config to /etc/ceph/ceph.conf 2024-06-15T00:47:31.266 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:47:31.266 INFO:teuthology.orchestra.run.smithi105.stdout:Creating mgr... 2024-06-15T00:47:31.266 INFO:teuthology.orchestra.run.smithi105.stdout:Verifying port 9283 ... 2024-06-15T00:47:31.439 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl reset-failed ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mgr.smithi105.oorfym 2024-06-15T00:47:31.439 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Failed to reset failed state of unit ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mgr.smithi105.oorfym.service: Unit ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mgr.smithi105.oorfym.service not loaded. 2024-06-15T00:47:31.583 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stderr Created symlink /etc/systemd/system/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968.target.wants/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mgr.smithi105.oorfym.service → /etc/systemd/system/ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@.service. 2024-06-15T00:47:31.916 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:31 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3306682296' entity='client.admin' 2024-06-15T00:47:32.087 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:47:32.087 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-06-15T00:47:32.095 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:47:32.095 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-06-15T00:47:32.095 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-06-15T00:47:32.095 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to enable service . firewalld.service is not available 2024-06-15T00:47:32.105 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:47:32.105 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-06-15T00:47:32.114 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:47:32.114 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-06-15T00:47:32.114 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-06-15T00:47:32.114 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to open ports <[9283]>. firewalld.service is not available 2024-06-15T00:47:32.114 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr to start... 2024-06-15T00:47:32.114 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr... 2024-06-15T00:47:32.905 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:32 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3081675929' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-15T00:47:33.208 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:33.211 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:47:33.211 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "bd0fff6c-2ab0-11ef-bc9d-c7b262605968", 2024-06-15T00:47:33.211 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-15T00:47:33.212 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "smithi105" 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 2, 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-15T00:47:33.213 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-15T00:47:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-15T00:47:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-15T00:47:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-15T00:47:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-15T00:47:33.221 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-15T00:47:33.222 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:33.223 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-06-15T00:47:25.568233+0000", 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf1e67c640 1 Processor -- start 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf1e67c640 1 -- start start 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf1e67c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf18104680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf1e67c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf18104bc0 con 0x7faf18075c80 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf17fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf18104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf17fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf18104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60472/0 (socket says 172.21.15.105:60472) 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.537+0000 7faf17fff640 1 -- 172.21.15.105:0/4078244953 learned_addr learned my addr 172.21.15.105:0/4078244953 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.538+0000 7faf17fff640 1 -- 172.21.15.105:0/4078244953 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf18104d00 con 0x7faf18075c80 2024-06-15T00:47:33.224 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.538+0000 7faf17fff640 1 --2- 172.21.15.105:0/4078244953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf18104680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faf08009840 tx=0x7faf080091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=7ff39fc0a72dbc48 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf16ffd640 1 -- 172.21.15.105:0/4078244953 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf08032440 con 0x7faf18075c80 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf16ffd640 1 -- 172.21.15.105:0/4078244953 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7faf08032a10 con 0x7faf18075c80 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf16ffd640 1 -- 172.21.15.105:0/4078244953 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf08031a00 con 0x7faf18075c80 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 -- 172.21.15.105:0/4078244953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 msgr2=0x7faf18104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 --2- 172.21.15.105:0/4078244953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf18104680 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7faf08009840 tx=0x7faf080091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 -- 172.21.15.105:0/4078244953 shutdown_connections 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 --2- 172.21.15.105:0/4078244953 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf18104680 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 -- 172.21.15.105:0/4078244953 >> 172.21.15.105:0/4078244953 conn(0x7faf180fc150 msgr2=0x7faf180fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 -- 172.21.15.105:0/4078244953 shutdown_connections 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.539+0000 7faf1e67c640 1 -- 172.21.15.105:0/4078244953 wait complete. 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf1e67c640 1 Processor -- start 2024-06-15T00:47:33.225 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf1e67c640 1 -- start start 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf1e67c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf181108a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf1e67c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf18110de0 con 0x7faf18075c80 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf17fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf181108a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf17fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf181108a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60486/0 (socket says 172.21.15.105:60486) 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.540+0000 7faf17fff640 1 -- 172.21.15.105:0/3081675929 learned_addr learned my addr 172.21.15.105:0/3081675929 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf17fff640 1 -- 172.21.15.105:0/3081675929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf08009450 con 0x7faf18075c80 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf17fff640 1 --2- 172.21.15.105:0/3081675929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf181108a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faf080098a0 tx=0x7faf08008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf0803d450 con 0x7faf18075c80 2024-06-15T00:47:33.226 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7faf0803da20 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf18111040 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf0803b860 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf18111b70 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7faf08039070 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.541+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7faf080466c0 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.542+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf1805ead0 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.544+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7faf08065790 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.607+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7faf181059f0 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.607+0000 7faf14ff9640 1 -- 172.21.15.105:0/3081675929 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7faf08054020 con 0x7faf18075c80 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.608+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 msgr2=0x7faf181108a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:33.227 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.608+0000 7faf1e67c640 1 --2- 172.21.15.105:0/3081675929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf181108a0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7faf080098a0 tx=0x7faf08008040 comp rx=0 tx=0).stop 2024-06-15T00:47:33.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.608+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 shutdown_connections 2024-06-15T00:47:33.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.608+0000 7faf1e67c640 1 --2- 172.21.15.105:0/3081675929 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7faf18075c80 0x7faf181108a0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:33.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.608+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 >> 172.21.15.105:0/3081675929 conn(0x7faf180fc150 msgr2=0x7faf180fcd80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:33.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.609+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 shutdown_connections 2024-06-15T00:47:33.228 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:32.609+0000 7faf1e67c640 1 -- 172.21.15.105:0/3081675929 wait complete. 2024-06-15T00:47:33.228 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (1/15)... 2024-06-15T00:47:36.012 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:35 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/457970677' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-15T00:47:36.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:36.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:47:36.291 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "bd0fff6c-2ab0-11ef-bc9d-c7b262605968", 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "smithi105" 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 5, 2024-06-15T00:47:36.292 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.293 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-15T00:47:36.294 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": false, 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:36.295 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-06-15T00:47:25.568233+0000", 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfbc441640 1 Processor -- start 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfbc441640 1 -- start start 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfbc441640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb4104680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfbc441640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb4104bc0 con 0x7fcfb4075c80 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfba1dd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb4104680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfba1dd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb4104680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60200/0 (socket says 172.21.15.105:60200) 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.642+0000 7fcfba1dd640 1 -- 172.21.15.105:0/3665253449 learned_addr learned my addr 172.21.15.105:0/3665253449 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.643+0000 7fcfba1dd640 1 -- 172.21.15.105:0/3665253449 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb4104d00 con 0x7fcfb4075c80 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.643+0000 7fcfba1dd640 1 --2- 172.21.15.105:0/3665253449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb4104680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009840 tx=0x7fcfb00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=35a40774f13bc124 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:36.296 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.644+0000 7fcfb91db640 1 -- 172.21.15.105:0/3665253449 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb0032440 con 0x7fcfb4075c80 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.644+0000 7fcfb91db640 1 -- 172.21.15.105:0/3665253449 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcfb0032a10 con 0x7fcfb4075c80 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.644+0000 7fcfb91db640 1 -- 172.21.15.105:0/3665253449 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb0031a00 con 0x7fcfb4075c80 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.644+0000 7fcfbc441640 1 -- 172.21.15.105:0/3665253449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 msgr2=0x7fcfb4104680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.644+0000 7fcfbc441640 1 --2- 172.21.15.105:0/3665253449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb4104680 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fcfb0009840 tx=0x7fcfb00091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 -- 172.21.15.105:0/3665253449 shutdown_connections 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 --2- 172.21.15.105:0/3665253449 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb4104680 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 -- 172.21.15.105:0/3665253449 >> 172.21.15.105:0/3665253449 conn(0x7fcfb40fc150 msgr2=0x7fcfb40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 -- 172.21.15.105:0/3665253449 shutdown_connections 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 -- 172.21.15.105:0/3665253449 wait complete. 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 Processor -- start 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.645+0000 7fcfbc441640 1 -- start start 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfbc441640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb410e670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfbc441640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcfb410ebb0 con 0x7fcfb4075c80 2024-06-15T00:47:36.297 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfba1dd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb410e670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfba1dd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb410e670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60216/0 (socket says 172.21.15.105:60216) 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfba1dd640 1 -- 172.21.15.105:0/457970677 learned_addr learned my addr 172.21.15.105:0/457970677 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfba1dd640 1 -- 172.21.15.105:0/457970677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcfb0009450 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfba1dd640 1 --2- 172.21.15.105:0/457970677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb410e670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00098a0 tx=0x7fcfb0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb003d450 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fcfb003da20 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcfb410ee10 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcfb003b860 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.646+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcfb410f940 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.647+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 1) v1 ==== 529+0+0 (secure 0 0 0) 0x7fcfb0039070 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.647+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7fcfb0046680 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.647+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcfb405ead0 con 0x7fcfb4075c80 2024-06-15T00:47:36.298 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.649+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+71547 (secure 0 0 0) 0x7fcfb0065790 con 0x7fcfb4075c80 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.712+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7fcfb4100e40 con 0x7fcfb4075c80 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.712+0000 7fcfaaffd640 1 -- 172.21.15.105:0/457970677 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1248 (secure 0 0 0) 0x7fcfb0054020 con 0x7fcfb4075c80 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.713+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 msgr2=0x7fcfb410e670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.713+0000 7fcfbc441640 1 --2- 172.21.15.105:0/457970677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb410e670 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fcfb00098a0 tx=0x7fcfb0008040 comp rx=0 tx=0).stop 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.713+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 shutdown_connections 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.713+0000 7fcfbc441640 1 --2- 172.21.15.105:0/457970677 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcfb4075c80 0x7fcfb410e670 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.713+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 >> 172.21.15.105:0/457970677 conn(0x7fcfb40fc150 msgr2=0x7fcfb4101ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.714+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 shutdown_connections 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:35.714+0000 7fcfbc441640 1 -- 172.21.15.105:0/457970677 wait complete. 2024-06-15T00:47:36.299 INFO:teuthology.orchestra.run.smithi105.stdout:mgr not available, waiting (2/15)... 2024-06-15T00:47:37.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: Activating manager daemon smithi105.oorfym 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: mgrmap e2: smithi105.oorfym(active, starting, since 0.00269906s) 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi105.oorfym", "id": "smithi105.oorfym"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: Manager daemon smithi105.oorfym is now available 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/trash_purge_schedule"}]: dispatch 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:36 smithi105 ceph-mon[28877]: from='mgr.14100 172.21.15.105:0/3801923671' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:38.671 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:38 smithi105 ceph-mon[28877]: mgrmap e3: smithi105.oorfym(active, since 1.00643s) 2024-06-15T00:47:39.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:39.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:47:39.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsid": "bd0fff6c-2ab0-11ef-bc9d-c7b262605968", 2024-06-15T00:47:39.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "health": { 2024-06-15T00:47:39.434 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "status": "HEALTH_OK", 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "checks": {}, 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mutes": [] 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "election_epoch": 5, 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum": [ 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 0 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_names": [ 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "smithi105" 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "quorum_age": 8, 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "monmap": { 2024-06-15T00:47:39.435 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "min_mon_release_name": "quincy", 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_mons": 1 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osdmap": { 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_osds": 0, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_up_osds": 0, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_up_since": 0, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_in_osds": 0, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "osd_in_since": 0, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_remapped_pgs": 0 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.436 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgmap": { 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "pgs_by_state": [], 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pgs": 0, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_pools": 0, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_objects": 0, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "data_bytes": 0, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_used": 0, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_avail": 0, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "bytes_total": 0 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "fsmap": { 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "by_rank": [], 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "up:standby": 0 2024-06-15T00:47:39.437 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap": { 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standbys": 0, 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modules": [ 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "iostat", 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "nfs", 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "restful" 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ], 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "servicemap": { 2024-06-15T00:47:39.438 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 1, 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "modified": "2024-06-15T00:47:25.568233+0000", 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "services": {} 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout }, 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "progress_events": {} 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.671+0000 7f0bb2b65640 1 Processor -- start 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.671+0000 7f0bb2b65640 1 -- start start 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.671+0000 7f0bb2b65640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.671+0000 7f0bb2b65640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac102440 con 0x7f0bac101af0 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.672+0000 7f0bb0901640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.672+0000 7f0bb0901640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60308/0 (socket says 172.21.15.105:60308) 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.672+0000 7f0bb0901640 1 -- 172.21.15.105:0/1228031313 learned_addr learned my addr 172.21.15.105:0/1228031313 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.672+0000 7f0bb0901640 1 -- 172.21.15.105:0/1228031313 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bac102580 con 0x7f0bac101af0 2024-06-15T00:47:39.439 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.673+0000 7f0bb0901640 1 --2- 172.21.15.105:0/1228031313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac101f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009840 tx=0x7f0ba00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=96d6da0b8bb15017 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.673+0000 7f0bab7fe640 1 -- 172.21.15.105:0/1228031313 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba0032440 con 0x7f0bac101af0 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.673+0000 7f0bab7fe640 1 -- 172.21.15.105:0/1228031313 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0ba0032a10 con 0x7f0bac101af0 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.673+0000 7f0bab7fe640 1 -- 172.21.15.105:0/1228031313 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba0031a00 con 0x7f0bac101af0 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1228031313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 msgr2=0x7f0bac101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 --2- 172.21.15.105:0/1228031313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac101f00 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0ba0009840 tx=0x7f0ba00091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1228031313 shutdown_connections 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 --2- 172.21.15.105:0/1228031313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac101f00 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1228031313 >> 172.21.15.105:0/1228031313 conn(0x7f0bac0fd2d0 msgr2=0x7f0bac0ff6e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1228031313 shutdown_connections 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1228031313 wait complete. 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 Processor -- start 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.674+0000 7f0bb2b65640 1 -- start start 2024-06-15T00:47:39.440 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb2b65640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac077c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb2b65640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bac078190 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb0901640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac077c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb0901640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac077c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60320/0 (socket says 172.21.15.105:60320) 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb0901640 1 -- 172.21.15.105:0/1317043467 learned_addr learned my addr 172.21.15.105:0/1317043467 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb0901640 1 -- 172.21.15.105:0/1317043467 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ba0009450 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb0901640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac077c50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00313e0 tx=0x7f0ba0008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba003d450 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f0ba003da20 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.675+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bac076220 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.676+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bac076690 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.676+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ba003b8d0 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.676+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f0ba0039070 con 0x7f0bac101af0 2024-06-15T00:47:39.441 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.677+0000 7f0ba97fa640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f0b9c0347c0 0x7f0b9c036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.677+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f0ba003d670 con 0x7f0bac101af0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.677+0000 7f0babfff640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f0b9c0347c0 0x7f0b9c036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.677+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b90005240 con 0x7f0bac101af0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.677+0000 7f0babfff640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f0b9c0347c0 0x7f0b9c036c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b98006fd0 tx=0x7f0b98006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.682+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0ba0041020 con 0x7f0bac101af0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.910+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f0b900054f0 con 0x7f0bac101af0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.910+0000 7f0ba97fa640 1 -- 172.21.15.105:0/1317043467 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1247 (secure 0 0 0) 0x7f0ba0037070 con 0x7f0bac101af0 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f0b9c0347c0 msgr2=0x7f0b9c036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f0b9c0347c0 0x7f0b9c036c70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f0b98006fd0 tx=0x7f0b98006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 msgr2=0x7f0bac077c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac077c50 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f0ba00313e0 tx=0x7f0ba0008040 comp rx=0 tx=0).stop 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 shutdown_connections 2024-06-15T00:47:39.442 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f0b9c0347c0 0x7f0b9c036c70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:39.443 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 --2- 172.21.15.105:0/1317043467 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0bac101af0 0x7f0bac077c50 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:39.443 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.912+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 >> 172.21.15.105:0/1317043467 conn(0x7f0bac0fd2d0 msgr2=0x7f0bac0fde80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:39.443 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.913+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 shutdown_connections 2024-06-15T00:47:39.443 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:38.913+0000 7f0bb2b65640 1 -- 172.21.15.105:0/1317043467 wait complete. 2024-06-15T00:47:39.443 INFO:teuthology.orchestra.run.smithi105.stdout:mgr is available 2024-06-15T00:47:39.714 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:39 smithi105 ceph-mon[28877]: mgrmap e4: smithi105.oorfym(active, since 2s) 2024-06-15T00:47:39.714 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:39 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1317043467' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-06-15T00:47:40.601 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [global] 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout fsid = bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_pg_remap = true 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_allow_primary_affinity = true 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_warn_on_no_sortbitwise = false 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_crush_chooseleaf_type = 0 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [mgr] 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mgr/telemetry/nag = false 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 2024-06-15T00:47:40.602 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout [osd] 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_backfillfull_ratio = 0.85 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_full_ratio = 0.9 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout mon_osd_nearfull_ratio = 0.8 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_map_max_advance = 10 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout osd_sloppy_crc = true 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729cd15640 1 Processor -- start 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729cd15640 1 -- start start 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729cd15640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f72980762c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729cd15640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72980780b0 con 0x7f7298077ca0 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f72980762c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f72980762c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60334/0 (socket says 172.21.15.105:60334) 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729659c640 1 -- 172.21.15.105:0/81261344 learned_addr learned my addr 172.21.15.105:0/81261344 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.788+0000 7f729659c640 1 -- 172.21.15.105:0/81261344 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7298076800 con 0x7f7298077ca0 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.789+0000 7f729659c640 1 --2- 172.21.15.105:0/81261344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f72980762c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7288009840 tx=0x7f72880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6c48f71ea3b959e server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:40.603 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.789+0000 7f729559a640 1 -- 172.21.15.105:0/81261344 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7288032440 con 0x7f7298077ca0 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.789+0000 7f729559a640 1 -- 172.21.15.105:0/81261344 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f7288032a10 con 0x7f7298077ca0 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.789+0000 7f729559a640 1 -- 172.21.15.105:0/81261344 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7288031a00 con 0x7f7298077ca0 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 -- 172.21.15.105:0/81261344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 msgr2=0x7f72980762c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 --2- 172.21.15.105:0/81261344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f72980762c0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f7288009840 tx=0x7f72880091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 -- 172.21.15.105:0/81261344 shutdown_connections 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 --2- 172.21.15.105:0/81261344 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f72980762c0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 -- 172.21.15.105:0/81261344 >> 172.21.15.105:0/81261344 conn(0x7f72980fd350 msgr2=0x7f72980ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 -- 172.21.15.105:0/81261344 shutdown_connections 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.790+0000 7f729cd15640 1 -- 172.21.15.105:0/81261344 wait complete. 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729cd15640 1 Processor -- start 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729cd15640 1 -- start start 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729cd15640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f7298112ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729cd15640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7298113010 con 0x7f7298077ca0 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f7298112ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:40.604 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f7298112ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60338/0 (socket says 172.21.15.105:60338) 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729659c640 1 -- 172.21.15.105:0/937692129 learned_addr learned my addr 172.21.15.105:0/937692129 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729659c640 1 -- 172.21.15.105:0/937692129 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7288009450 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729659c640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f7298112ad0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f72880098a0 tx=0x7f7288008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f728803d450 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f728803da20 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7298113210 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f728803b860 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.791+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7298113680 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.792+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f7288039070 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.792+0000 7f7286ffd640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f7280034810 0x7f7280036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.792+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f72880670c0 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.792+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f729805ead0 con 0x7f7298077ca0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.792+0000 7f7295d9b640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f7280034810 0x7f7280036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:40.605 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.793+0000 7f7295d9b640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f7280034810 0x7f7280036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.797+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7288041070 con 0x7f7298077ca0 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.961+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f729804b480 con 0x7f7298077ca0 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.962+0000 7f7286ffd640 1 -- 172.21.15.105:0/937692129 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v3) v1 ==== 70+0+373 (secure 0 0 0) 0x7f7288037030 con 0x7f7298077ca0 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.963+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f7280034810 msgr2=0x7f7280036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.963+0000 7f729cd15640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f7280034810 0x7f7280036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f728c006fd0 tx=0x7f728c006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.963+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 msgr2=0x7f7298112ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.963+0000 7f729cd15640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f7298112ad0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f72880098a0 tx=0x7f7288008040 comp rx=0 tx=0).stop 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.963+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 shutdown_connections 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.964+0000 7f729cd15640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f7280034810 0x7f7280036cc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.964+0000 7f729cd15640 1 --2- 172.21.15.105:0/937692129 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7298077ca0 0x7f7298112ad0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.964+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 >> 172.21.15.105:0/937692129 conn(0x7f72980fd350 msgr2=0x7f72980fdf30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.964+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 shutdown_connections 2024-06-15T00:47:40.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:39.964+0000 7f729cd15640 1 -- 172.21.15.105:0/937692129 wait complete. 2024-06-15T00:47:40.607 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling cephadm module... 2024-06-15T00:47:40.704 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:40 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/937692129' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-06-15T00:47:41.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:41 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/4119334357' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-06-15T00:47:42.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:42 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/4119334357' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-06-15T00:47:42.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:42 smithi105 ceph-mon[28877]: mgrmap e5: smithi105.oorfym(active, since 5s) 2024-06-15T00:47:45.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f296106c640 1 Processor -- start 2024-06-15T00:47:45.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f296106c640 1 -- start start 2024-06-15T00:47:45.000 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f296106c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c101f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f296106c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c102440 con 0x7f295c101af0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f295ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c101f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f295ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c101f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60344/0 (socket says 172.21.15.105:60344) 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f295ad9d640 1 -- 172.21.15.105:0/3197157068 learned_addr learned my addr 172.21.15.105:0/3197157068 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.941+0000 7f295ad9d640 1 -- 172.21.15.105:0/3197157068 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f295c102580 con 0x7f295c101af0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.942+0000 7f295ad9d640 1 --2- 172.21.15.105:0/3197157068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c101f00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f294c009840 tx=0x7f294c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=10973affaee81f06 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.942+0000 7f2959d9b640 1 -- 172.21.15.105:0/3197157068 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c032440 con 0x7f295c101af0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.942+0000 7f2959d9b640 1 -- 172.21.15.105:0/3197157068 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f294c032a10 con 0x7f295c101af0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.942+0000 7f2959d9b640 1 -- 172.21.15.105:0/3197157068 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c031a00 con 0x7f295c101af0 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- 172.21.15.105:0/3197157068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 msgr2=0x7f295c101f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 --2- 172.21.15.105:0/3197157068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c101f00 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7f294c009840 tx=0x7f294c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:45.001 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- 172.21.15.105:0/3197157068 shutdown_connections 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 --2- 172.21.15.105:0/3197157068 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c101f00 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- 172.21.15.105:0/3197157068 >> 172.21.15.105:0/3197157068 conn(0x7f295c0fd330 msgr2=0x7f295c0ff740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- 172.21.15.105:0/3197157068 shutdown_connections 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- 172.21.15.105:0/3197157068 wait complete. 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 Processor -- start 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- start start 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.943+0000 7f296106c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c110dc0 con 0x7f295c101af0 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f295ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f295ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c110880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60350/0 (socket says 172.21.15.105:60350) 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f295ad9d640 1 -- 172.21.15.105:0/4119334357 learned_addr learned my addr 172.21.15.105:0/4119334357 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:45.002 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f295ad9d640 1 -- 172.21.15.105:0/4119334357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c009450 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f295ad9d640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c110880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f294c0098a0 tx=0x7f294c008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c03d450 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f294c03da20 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c110fc0 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c03b860 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.944+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c111430 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.945+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 4) v1 ==== 41935+0+0 (secure 0 0 0) 0x7f294c039070 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.945+0000 7f29437fe640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f29440347c0 0x7f2944036c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.945+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 721+0+0 (secure 0 0 0) 0x7f294c067260 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.945+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c05ead0 con 0x7f295c101af0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.945+0000 7f295a59c640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f29440347c0 0x7f2944036c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.945+0000 7f295a59c640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f29440347c0 0x7f2944036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2950006fd0 tx=0x7f2950006db0 comp rx=0 tx=0).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:45.003 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:40.950+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f294c041020 con 0x7f295c101af0 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.157+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7f295c04b480 con 0x7f295c101af0 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.530+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 5) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f294c03dc00 con 0x7f295c101af0 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.531+0000 7f29437fe640 1 -- 172.21.15.105:0/4119334357 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7f294c039b60 con 0x7f295c101af0 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f29440347c0 msgr2=0x7f2944036c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f29440347c0 0x7f2944036c70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f2950006fd0 tx=0x7f2950006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 msgr2=0x7f295c110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c110880 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f294c0098a0 tx=0x7f294c008040 comp rx=0 tx=0).stop 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 shutdown_connections 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:6800/135592780,v1:172.21.15.105:6801/135592780] conn(0x7f29440347c0 0x7f2944036c70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 --2- 172.21.15.105:0/4119334357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f295c101af0 0x7f295c110880 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.533+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 >> 172.21.15.105:0/4119334357 conn(0x7f295c0fd330 msgr2=0x7f295c0fdec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.534+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 shutdown_connections 2024-06-15T00:47:45.004 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:41.534+0000 7f296106c640 1 -- 172.21.15.105:0/4119334357 wait complete. 2024-06-15T00:47:46.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: Active manager daemon smithi105.oorfym restarted 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: Activating manager daemon smithi105.oorfym 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: osdmap e2: 0 total, 0 up, 0 in 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: mgrmap e6: smithi105.oorfym(active, starting, since 0.00314048s) 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi105.oorfym", "id": "smithi105.oorfym"}]: dispatch 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: Manager daemon smithi105.oorfym is now available 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:46.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:47:46.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:47:46.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:47:46.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:47:46.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:46 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/trash_purge_schedule"}]: dispatch 2024-06-15T00:47:47.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:47 smithi105 ceph-mon[28877]: Found migration_current of "None". Setting to last migration. 2024-06-15T00:47:47.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:47 smithi105 ceph-mon[28877]: mgrmap e7: smithi105.oorfym(active, since 1.00527s) 2024-06-15T00:47:49.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: [15/Jun/2024:00:47:47] ENGINE Bus STARTING 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: [15/Jun/2024:00:47:47] ENGINE Serving on https://172.21.15.105:7150 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: [15/Jun/2024:00:47:47] ENGINE Bus STARTED 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: [15/Jun/2024:00:47:47] ENGINE Error in HTTPServer.serve 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Traceback (most recent call last): 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._run(expiration_interval) 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:47:49.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: s = self.context.wrap_socket( 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: return self.sslsocket_class._create( 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self.do_handshake() 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._sslobj.do_handshake() 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:47:49.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:48 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2867302666' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-15T00:47:50.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:49 smithi105 ceph-mon[28877]: mgrmap e8: smithi105.oorfym(active, since 2s) 2024-06-15T00:47:53.542 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 7, 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "active_name": "smithi105.oorfym", 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.144+0000 7f999687d640 1 Processor -- start 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.144+0000 7f999687d640 1 -- start start 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.144+0000 7f999687d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990101f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.144+0000 7f999687d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99901024a0 con 0x7f9990101b50 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.145+0000 7f998ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:53.543 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.145+0000 7f998ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990101f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40376/0 (socket says 172.21.15.105:40376) 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.145+0000 7f998ffff640 1 -- 172.21.15.105:0/1352134289 learned_addr learned my addr 172.21.15.105:0/1352134289 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.145+0000 7f998ffff640 1 -- 172.21.15.105:0/1352134289 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99901025e0 con 0x7f9990101b50 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.145+0000 7f998ffff640 1 --2- 172.21.15.105:0/1352134289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990101f60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9980009840 tx=0x7f99800091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b69ef0c61f29b3bf server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f998f7fe640 1 -- 172.21.15.105:0/1352134289 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980032440 con 0x7f9990101b50 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f998f7fe640 1 -- 172.21.15.105:0/1352134289 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f9980032a10 con 0x7f9990101b50 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f998f7fe640 1 -- 172.21.15.105:0/1352134289 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9980031a00 con 0x7f9990101b50 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 -- 172.21.15.105:0/1352134289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 msgr2=0x7f9990101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 --2- 172.21.15.105:0/1352134289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990101f60 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 crypto rx=0x7f9980009840 tx=0x7f99800091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 -- 172.21.15.105:0/1352134289 shutdown_connections 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 --2- 172.21.15.105:0/1352134289 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990101f60 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 -- 172.21.15.105:0/1352134289 >> 172.21.15.105:0/1352134289 conn(0x7f99900fd350 msgr2=0x7f99900ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:53.544 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 -- 172.21.15.105:0/1352134289 shutdown_connections 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.146+0000 7f999687d640 1 -- 172.21.15.105:0/1352134289 wait complete. 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f999687d640 1 Processor -- start 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f999687d640 1 -- start start 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f999687d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f999687d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9990110dd0 con 0x7f9990101b50 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f998ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f998ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40386/0 (socket says 172.21.15.105:40386) 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f998ffff640 1 -- 172.21.15.105:0/2867302666 learned_addr learned my addr 172.21.15.105:0/2867302666 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.147+0000 7f998ffff640 1 -- 172.21.15.105:0/2867302666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9980009450 con 0x7f9990101b50 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f998ffff640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990110890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f99800098a0 tx=0x7f9980008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998003d450 con 0x7f9990101b50 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f998003da20 con 0x7f9990101b50 2024-06-15T00:47:53.545 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9990110fd0 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f998003b860 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9990111440 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.148+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 7) v1 ==== 41840+0+0 (secure 0 0 0) 0x7f9980039070 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.149+0000 7f998d7fa640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f997c034730 0x7f997c036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.149+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f99800672a0 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.149+0000 7f9987fff640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f997c034730 0x7f997c036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.149+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f999005ead0 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.149+0000 7f9987fff640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f997c034730 0x7f997c036be0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.154+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9980041020 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.364+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f9990111c60 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.365+0000 7f998d7fa640 1 -- 172.21.15.105:0/2867302666 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v7) v1 ==== 56+0+103 (secure 0 0 0) 0x7f9980037070 con 0x7f9990101b50 2024-06-15T00:47:53.546 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f997c034730 msgr2=0x7f997c036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f997c034730 0x7f997c036be0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7f9978006fd0 tx=0x7f9978006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 msgr2=0x7f9990110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990110890 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7f99800098a0 tx=0x7f9980008040 comp rx=0 tx=0).stop 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 shutdown_connections 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f997c034730 0x7f997c036be0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 --2- 172.21.15.105:0/2867302666 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9990101b50 0x7f9990110890 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 >> 172.21.15.105:0/2867302666 conn(0x7f99900fd350 msgr2=0x7f99900fdef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 shutdown_connections 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:48.367+0000 7f999687d640 1 -- 172.21.15.105:0/2867302666 wait complete. 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for the mgr to restart... 2024-06-15T00:47:53.547 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr epoch 7... 2024-06-15T00:47:54.573 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:47:54.573 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 8, 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.941+0000 7f6ad3d82640 1 Processor -- start 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.941+0000 7f6ad3d82640 1 -- start start 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.941+0000 7f6ad3d82640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc104a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.941+0000 7f6ad3d82640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc075e20 con 0x7f6acc102650 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.941+0000 7f6ad1b1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc104a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.941+0000 7f6ad1b1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc104a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33044/0 (socket says 172.21.15.105:33044) 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.942+0000 7f6ad1b1e640 1 -- 172.21.15.105:0/3749950342 learned_addr learned my addr 172.21.15.105:0/3749950342 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.942+0000 7f6ad1b1e640 1 -- 172.21.15.105:0/3749950342 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6acc075f60 con 0x7f6acc102650 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.942+0000 7f6ad1b1e640 1 --2- 172.21.15.105:0/3749950342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc104a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009840 tx=0x7f6ac80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=c991da75e6f6e75c server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:54.574 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad0b1c640 1 -- 172.21.15.105:0/3749950342 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ac8032440 con 0x7f6acc102650 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad0b1c640 1 -- 172.21.15.105:0/3749950342 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6ac8032a10 con 0x7f6acc102650 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad0b1c640 1 -- 172.21.15.105:0/3749950342 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ac8031a00 con 0x7f6acc102650 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 -- 172.21.15.105:0/3749950342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 msgr2=0x7f6acc104a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 --2- 172.21.15.105:0/3749950342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc104a30 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7f6ac8009840 tx=0x7f6ac80091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 -- 172.21.15.105:0/3749950342 shutdown_connections 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 --2- 172.21.15.105:0/3749950342 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc104a30 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 -- 172.21.15.105:0/3749950342 >> 172.21.15.105:0/3749950342 conn(0x7f6acc0fc130 msgr2=0x7f6acc0fe540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 -- 172.21.15.105:0/3749950342 shutdown_connections 2024-06-15T00:47:54.575 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.943+0000 7f6ad3d82640 1 -- 172.21.15.105:0/3749950342 wait complete. 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad3d82640 1 Processor -- start 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad3d82640 1 -- start start 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad3d82640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc110880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad3d82640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6acc110dc0 con 0x7f6acc102650 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad1b1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc110880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad1b1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc110880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33054/0 (socket says 172.21.15.105:33054) 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad1b1e640 1 -- 172.21.15.105:0/1870618029 learned_addr learned my addr 172.21.15.105:0/1870618029 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.944+0000 7f6ad1b1e640 1 -- 172.21.15.105:0/1870618029 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ac8009450 con 0x7f6acc102650 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.945+0000 7f6ad1b1e640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc110880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80098a0 tx=0x7f6ac8008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:54.576 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.945+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ac803d450 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.945+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6acc110fc0 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.945+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f6ac803da20 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.945+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ac803b860 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.945+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6acc111430 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.946+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f6ac8039070 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.946+0000 7f6ac27fc640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f6ab8034810 0x7f6ab8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.946+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6ab80373d0 con 0x7f6ab8034810 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.946+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f6ac8067340 con 0x7f6acc102650 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.946+0000 7f6ad131d640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f6ab8034810 0x7f6ab8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.946+0000 7f6ad131d640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f6ab8034810 0x7f6ab8036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6abc006fd0 tx=0x7f6abc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.947+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7f6ab80373d0 con 0x7f6ab8034810 2024-06-15T00:47:54.577 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f6acc075f60 con 0x7f6ab8034810 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ac27fc640 1 -- 172.21.15.105:0/1870618029 <== mgr.14118 v2:172.21.15.105:6800/2469050433 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f6acc075f60 con 0x7f6ab8034810 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f6ab8034810 msgr2=0x7f6ab8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f6ab8034810 0x7f6ab8036cc0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f6abc006fd0 tx=0x7f6abc006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 msgr2=0x7f6acc110880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc110880 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7f6ac80098a0 tx=0x7f6ac8008040 comp rx=0 tx=0).stop 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 shutdown_connections 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f6ab8034810 0x7f6ab8036cc0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 --2- 172.21.15.105:0/1870618029 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6acc102650 0x7f6acc110880 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 >> 172.21.15.105:0/1870618029 conn(0x7f6acc0fc130 msgr2=0x7f6acc0fccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 shutdown_connections 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:53.952+0000 7f6ad3d82640 1 -- 172.21.15.105:0/1870618029 wait complete. 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:mgr epoch 7 is available 2024-06-15T00:47:54.578 INFO:teuthology.orchestra.run.smithi105.stdout:Setting orchestrator backend to cephadm... 2024-06-15T00:47:54.867 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:54 smithi105 ceph-mon[28877]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-15T00:47:54.867 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:54 smithi105 ceph-mon[28877]: from='client.14128 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe097be640 1 Processor -- start 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe097be640 1 -- start start 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe097be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe041046e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe097be640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe04104c20 con 0x7fbe04075c20 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe02ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe041046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe02ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe041046e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33068/0 (socket says 172.21.15.105:33068) 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe02ffd640 1 -- 172.21.15.105:0/592293984 learned_addr learned my addr 172.21.15.105:0/592293984 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.943+0000 7fbe02ffd640 1 -- 172.21.15.105:0/592293984 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe04104d60 con 0x7fbe04075c20 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.944+0000 7fbe02ffd640 1 --2- 172.21.15.105:0/592293984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe041046e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009840 tx=0x7fbdf40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=162127f5dd08c776 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:55.720 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.944+0000 7fbe01ffb640 1 -- 172.21.15.105:0/592293984 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4032440 con 0x7fbe04075c20 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.944+0000 7fbe01ffb640 1 -- 172.21.15.105:0/592293984 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbdf4032a10 con 0x7fbe04075c20 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.944+0000 7fbe01ffb640 1 -- 172.21.15.105:0/592293984 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4031a00 con 0x7fbe04075c20 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 -- 172.21.15.105:0/592293984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 msgr2=0x7fbe041046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 --2- 172.21.15.105:0/592293984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe041046e0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4009840 tx=0x7fbdf40091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 -- 172.21.15.105:0/592293984 shutdown_connections 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 --2- 172.21.15.105:0/592293984 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe041046e0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 -- 172.21.15.105:0/592293984 >> 172.21.15.105:0/592293984 conn(0x7fbe040fc150 msgr2=0x7fbe040fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 -- 172.21.15.105:0/592293984 shutdown_connections 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 -- 172.21.15.105:0/592293984 wait complete. 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.945+0000 7fbe097be640 1 Processor -- start 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe097be640 1 -- start start 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe097be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe04102010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe097be640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe04102550 con 0x7fbe04075c20 2024-06-15T00:47:55.721 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe02ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe04102010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe02ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe04102010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33082/0 (socket says 172.21.15.105:33082) 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe02ffd640 1 -- 172.21.15.105:0/173662550 learned_addr learned my addr 172.21.15.105:0/173662550 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe02ffd640 1 -- 172.21.15.105:0/173662550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdf4009450 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe02ffd640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe04102010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4006f00 tx=0x7fbdf4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf4031e20 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbdf403d8c0 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe041005e0 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdf403b840 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.946+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe04100a50 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.947+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fbdf4039070 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.948+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe0404b480 con 0x7fbe04075c20 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.948+0000 7fbdeb7fe640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbdec034810 0x7fbdec036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:55.722 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.948+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbdf40677e0 con 0x7fbe04075c20 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.948+0000 7fbe027fc640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbdec034810 0x7fbdec036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.948+0000 7fbe027fc640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbdec034810 0x7fbdec036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8006fd0 tx=0x7fbdf8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:54.952+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbdf403c7c0 con 0x7fbe04075c20 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.125+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe04101200 con 0x7fbdec034810 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.133+0000 7fbdeb7fe640 1 -- 172.21.15.105:0/173662550 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fbe04101200 con 0x7fbdec034810 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbdec034810 msgr2=0x7fbdec036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbdec034810 0x7fbdec036cc0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7fbdf8006fd0 tx=0x7fbdf8006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 msgr2=0x7fbe04102010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe04102010 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 crypto rx=0x7fbdf4006f00 tx=0x7fbdf4008040 comp rx=0 tx=0).stop 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 shutdown_connections 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbdec034810 0x7fbdec036cc0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 --2- 172.21.15.105:0/173662550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe04075c20 0x7fbe04102010 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:55.723 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.135+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 >> 172.21.15.105:0/173662550 conn(0x7fbe040fc150 msgr2=0x7fbe040fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:55.724 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.136+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 shutdown_connections 2024-06-15T00:47:55.724 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:55.136+0000 7fbe097be640 1 -- 172.21.15.105:0/173662550 wait complete. 2024-06-15T00:47:56.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:56 smithi105 ceph-mon[28877]: from='client.14130 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:47:56.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:56 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:56 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:47:56.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout value unchanged 2024-06-15T00:47:56.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.131+0000 7f3b7c334640 1 Processor -- start 2024-06-15T00:47:56.792 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.131+0000 7f3b7c334640 1 -- start start 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.131+0000 7f3b7c334640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.131+0000 7f3b7c334640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b74104c00 con 0x7f3b74075c40 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.131+0000 7f3b7a0d0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.132+0000 7f3b7a0d0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33088/0 (socket says 172.21.15.105:33088) 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.132+0000 7f3b7a0d0640 1 -- 172.21.15.105:0/3565984490 learned_addr learned my addr 172.21.15.105:0/3565984490 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.132+0000 7f3b7a0d0640 1 -- 172.21.15.105:0/3565984490 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b74104d40 con 0x7f3b74075c40 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.132+0000 7f3b7a0d0640 1 --2- 172.21.15.105:0/3565984490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741046c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009840 tx=0x7f3b700091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b33491a9b3bede27 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b790ce640 1 -- 172.21.15.105:0/3565984490 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b70032440 con 0x7f3b74075c40 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b790ce640 1 -- 172.21.15.105:0/3565984490 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3b70032a10 con 0x7f3b74075c40 2024-06-15T00:47:56.793 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b790ce640 1 -- 172.21.15.105:0/3565984490 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b70031a00 con 0x7f3b74075c40 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 -- 172.21.15.105:0/3565984490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 msgr2=0x7f3b741046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 --2- 172.21.15.105:0/3565984490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741046c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 crypto rx=0x7f3b70009840 tx=0x7f3b700091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 -- 172.21.15.105:0/3565984490 shutdown_connections 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 --2- 172.21.15.105:0/3565984490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741046c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 -- 172.21.15.105:0/3565984490 >> 172.21.15.105:0/3565984490 conn(0x7f3b740fc150 msgr2=0x7f3b740fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 -- 172.21.15.105:0/3565984490 shutdown_connections 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.133+0000 7f3b7c334640 1 -- 172.21.15.105:0/3565984490 wait complete. 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7c334640 1 Processor -- start 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7c334640 1 -- start start 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7c334640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741108c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7c334640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b74110e00 con 0x7f3b74075c40 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7a0d0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:56.794 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7a0d0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741108c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33096/0 (socket says 172.21.15.105:33096) 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7a0d0640 1 -- 172.21.15.105:0/471810723 learned_addr learned my addr 172.21.15.105:0/471810723 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.134+0000 7f3b7a0d0640 1 -- 172.21.15.105:0/471810723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b70009450 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.135+0000 7f3b7a0d0640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741108c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b70006f00 tx=0x7f3b70008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.135+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b70031e20 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.135+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f3b7003d8c0 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.135+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b74111000 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.135+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3b7003b840 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.135+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b74111470 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.136+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f3b70039070 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.136+0000 7f3b6affd640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3b60034810 0x7f3b60036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.136+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f3b70067610 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.136+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b7405ead0 con 0x7f3b74075c40 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.136+0000 7f3b798cf640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3b60034810 0x7f3b60036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:56.795 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.136+0000 7f3b798cf640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3b60034810 0x7f3b60036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b64006fd0 tx=0x7f3b64006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.140+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3b70041070 con 0x7f3b74075c40 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.309+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f3b740fd5b0 con 0x7f3b60034810 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.310+0000 7f3b6affd640 1 -- 172.21.15.105:0/471810723 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f3b740fd5b0 con 0x7f3b60034810 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3b60034810 msgr2=0x7f3b60036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3b60034810 0x7f3b60036cc0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f3b64006fd0 tx=0x7f3b64006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 msgr2=0x7f3b741108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741108c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 crypto rx=0x7f3b70006f00 tx=0x7f3b70008040 comp rx=0 tx=0).stop 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 shutdown_connections 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3b60034810 0x7f3b60036cc0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 --2- 172.21.15.105:0/471810723 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3b74075c40 0x7f3b741108c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 >> 172.21.15.105:0/471810723 conn(0x7f3b740fc150 msgr2=0x7f3b740fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 shutdown_connections 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:56.312+0000 7f3b7c334640 1 -- 172.21.15.105:0/471810723 wait complete. 2024-06-15T00:47:56.796 INFO:teuthology.orchestra.run.smithi105.stdout:Generating ssh key... 2024-06-15T00:47:57.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:57 smithi105 ceph-mon[28877]: from='client.14132 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:47:58.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:58 smithi105 ceph-mon[28877]: from='client.14134 -' entity='client.admin' cmd=[{"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:47:58.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:58 smithi105 ceph-mon[28877]: Generating ssh key... 2024-06-15T00:47:58.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:58 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:58.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:58 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.186+0000 7ff0d9445640 1 Processor -- start 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.186+0000 7ff0d9445640 1 -- start start 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.186+0000 7ff0d9445640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41046e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.186+0000 7ff0d9445640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d4104c20 con 0x7ff0d4075c20 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.187+0000 7ff0d2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41046e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.187+0000 7ff0d2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41046e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33102/0 (socket says 172.21.15.105:33102) 2024-06-15T00:47:58.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.187+0000 7ff0d2ffd640 1 -- 172.21.15.105:0/3995384546 learned_addr learned my addr 172.21.15.105:0/3995384546 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.187+0000 7ff0d2ffd640 1 -- 172.21.15.105:0/3995384546 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0d4104d60 con 0x7ff0d4075c20 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.187+0000 7ff0d2ffd640 1 --2- 172.21.15.105:0/3995384546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41046e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009840 tx=0x7ff0c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=74538354dc5a3523 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d1ffb640 1 -- 172.21.15.105:0/3995384546 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c4032440 con 0x7ff0d4075c20 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d1ffb640 1 -- 172.21.15.105:0/3995384546 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff0c4032a10 con 0x7ff0d4075c20 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d1ffb640 1 -- 172.21.15.105:0/3995384546 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c4031a00 con 0x7ff0d4075c20 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d9445640 1 -- 172.21.15.105:0/3995384546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 msgr2=0x7ff0d41046e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d9445640 1 --2- 172.21.15.105:0/3995384546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41046e0 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7ff0c4009840 tx=0x7ff0c40091d0 comp rx=0 tx=0).stop 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d9445640 1 -- 172.21.15.105:0/3995384546 shutdown_connections 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d9445640 1 --2- 172.21.15.105:0/3995384546 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41046e0 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.188+0000 7ff0d9445640 1 -- 172.21.15.105:0/3995384546 >> 172.21.15.105:0/3995384546 conn(0x7ff0d40fc150 msgr2=0x7ff0d40fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d9445640 1 -- 172.21.15.105:0/3995384546 shutdown_connections 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d9445640 1 -- 172.21.15.105:0/3995384546 wait complete. 2024-06-15T00:47:58.200 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d9445640 1 Processor -- start 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d9445640 1 -- start start 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d9445640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41108b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d9445640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0d4110df0 con 0x7ff0d4075c20 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.189+0000 7ff0d2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41108b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0d2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41108b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33116/0 (socket says 172.21.15.105:33116) 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0d2ffd640 1 -- 172.21.15.105:0/3949881112 learned_addr learned my addr 172.21.15.105:0/3949881112 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0d2ffd640 1 -- 172.21.15.105:0/3949881112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c4009450 con 0x7ff0d4075c20 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0d2ffd640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41108b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40098a0 tx=0x7ff0c4008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c403d450 con 0x7ff0d4075c20 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7ff0c403da20 con 0x7ff0d4075c20 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0d4110ff0 con 0x7ff0d4075c20 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff0c403b860 con 0x7ff0d4075c20 2024-06-15T00:47:58.201 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.190+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0d4111460 con 0x7ff0d4075c20 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.191+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7ff0c4039070 con 0x7ff0d4075c20 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.191+0000 7ff0bb7fe640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7ff0bc034810 0x7ff0bc036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.191+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7ff0c40674d0 con 0x7ff0d4075c20 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.191+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0d405ead0 con 0x7ff0d4075c20 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.191+0000 7ff0d27fc640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7ff0bc034810 0x7ff0bc036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.191+0000 7ff0d27fc640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7ff0bc034810 0x7ff0bc036cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8006fd0 tx=0x7ff0c8006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.195+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff0c4041070 con 0x7ff0d4075c20 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.363+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7ff0d40fd530 con 0x7ff0bc034810 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.603+0000 7ff0bb7fe640 1 -- 172.21.15.105:0/3949881112 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7ff0d40fd530 con 0x7ff0bc034810 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7ff0bc034810 msgr2=0x7ff0bc036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7ff0bc034810 0x7ff0bc036cc0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7ff0c8006fd0 tx=0x7ff0c8006db0 comp rx=0 tx=0).stop 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 msgr2=0x7ff0d41108b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:47:58.202 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41108b0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 crypto rx=0x7ff0c40098a0 tx=0x7ff0c4008040 comp rx=0 tx=0).stop 2024-06-15T00:47:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 shutdown_connections 2024-06-15T00:47:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7ff0bc034810 0x7ff0bc036cc0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 --2- 172.21.15.105:0/3949881112 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff0d4075c20 0x7ff0d41108b0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:47:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 >> 172.21.15.105:0/3949881112 conn(0x7ff0d40fc150 msgr2=0x7ff0d40fcd60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:47:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 shutdown_connections 2024-06-15T00:47:58.203 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:57.605+0000 7ff0d9445640 1 -- 172.21.15.105:0/3949881112 wait complete. 2024-06-15T00:48:00.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:47:59 smithi105 ceph-mon[28877]: from='client.14136 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:00.057 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCIshCpzeVToblkQEzSASMOxEMm1Rl+Zg2b/0FdhEI2Y63k2FeDjBRMyIDSHXGzUGFeWGUdHrgigi3N1N3C8Fe+9KpcmkstcJ/VqAexW/D0nR0O4Oj0DuWoxNlgQdOdJpsSlIDAX3+G/3B3m9lP3m7f78uwhVD3ZZtwkblq/YkrqumFU2z0EzQ0LER23Ex0DvmMq4JL89humV1jm6Oxc0xRZiFy8DrbSh2ak01IgaHPDAuxzQOxaUM2cf5o4VL9fJKSZt0dzZMxABh9mV/ifhmasRueazSDCmR2gFyHoUA6warz/fumszcvkaMiaNfJ9HzctPUZF/4+Dw++ErHbvQYtE5x2jyFZbVl9+OVAXmDe6r5FP2c0OdobS9Dkd73DzK3kqd9dgST3qfQkK51XGHyRL+9TSIstQ/+T1VM5IUKpoEMj+kRRrRiqBLKfl5jkqTvA54bEXV1g4Y1Aqc9X+BbBZMHjsnNZdhSKF+XSTdGwDboVJa9OSKw/mBn+Ve01s18= ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.544+0000 7f5c45df4640 1 Processor -- start 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.544+0000 7f5c45df4640 1 -- start start 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.545+0000 7f5c45df4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.545+0000 7f5c45df4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40104c00 con 0x7f5c40075c40 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.545+0000 7f5c3f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.545+0000 7f5c3f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33120/0 (socket says 172.21.15.105:33120) 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.545+0000 7f5c3f7fe640 1 -- 172.21.15.105:0/3366309898 learned_addr learned my addr 172.21.15.105:0/3366309898 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.545+0000 7f5c3f7fe640 1 -- 172.21.15.105:0/3366309898 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c40104d40 con 0x7f5c40075c40 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c3f7fe640 1 --2- 172.21.15.105:0/3366309898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401046c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009840 tx=0x7f5c300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3895c150fd3711c1 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c3effd640 1 -- 172.21.15.105:0/3366309898 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c30032440 con 0x7f5c40075c40 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c3effd640 1 -- 172.21.15.105:0/3366309898 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5c30032a10 con 0x7f5c40075c40 2024-06-15T00:48:00.058 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c3effd640 1 -- 172.21.15.105:0/3366309898 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c30031a00 con 0x7f5c40075c40 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 -- 172.21.15.105:0/3366309898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 msgr2=0x7f5c401046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 --2- 172.21.15.105:0/3366309898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401046c0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 crypto rx=0x7f5c30009840 tx=0x7f5c300091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 -- 172.21.15.105:0/3366309898 shutdown_connections 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 --2- 172.21.15.105:0/3366309898 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401046c0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 -- 172.21.15.105:0/3366309898 >> 172.21.15.105:0/3366309898 conn(0x7f5c400fc150 msgr2=0x7f5c400fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 -- 172.21.15.105:0/3366309898 shutdown_connections 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.546+0000 7f5c45df4640 1 -- 172.21.15.105:0/3366309898 wait complete. 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c45df4640 1 Processor -- start 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c45df4640 1 -- start start 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c45df4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401108c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c45df4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c40110e00 con 0x7f5c40075c40 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c3f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:00.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c3f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401108c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33128/0 (socket says 172.21.15.105:33128) 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c3f7fe640 1 -- 172.21.15.105:0/3635572058 learned_addr learned my addr 172.21.15.105:0/3635572058 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.547+0000 7f5c3f7fe640 1 -- 172.21.15.105:0/3635572058 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c30009450 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.548+0000 7f5c3f7fe640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401108c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5c300098a0 tx=0x7f5c30008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.548+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c3003d450 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.548+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f5c3003da20 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.548+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c3003b860 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.548+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c40111000 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.548+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c40111470 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.549+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f5c30039070 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.549+0000 7f5c3cff9640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f5c2c034860 0x7f5c2c036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.549+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f5c30067550 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.549+0000 7f5c37fff640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f5c2c034860 0x7f5c2c036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.549+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c4005ead0 con 0x7f5c40075c40 2024-06-15T00:48:00.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.549+0000 7f5c37fff640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f5c2c034860 0x7f5c2c036d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c28006fd0 tx=0x7f5c28006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.553+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5c30041020 con 0x7f5c40075c40 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.719+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f5c400fd5b0 con 0x7f5c2c034860 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.720+0000 7f5c3cff9640 1 -- 172.21.15.105:0/3635572058 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f5c400fd5b0 con 0x7f5c2c034860 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f5c2c034860 msgr2=0x7f5c2c036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f5c2c034860 0x7f5c2c036d10 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f5c28006fd0 tx=0x7f5c28006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 msgr2=0x7f5c401108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401108c0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 crypto rx=0x7f5c300098a0 tx=0x7f5c30008040 comp rx=0 tx=0).stop 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 shutdown_connections 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f5c2c034860 0x7f5c2c036d10 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 --2- 172.21.15.105:0/3635572058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5c40075c40 0x7f5c401108c0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 >> 172.21.15.105:0/3635572058 conn(0x7f5c400fc150 msgr2=0x7f5c400fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 shutdown_connections 2024-06-15T00:48:00.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:47:58.722+0000 7f5c45df4640 1 -- 172.21.15.105:0/3635572058 wait complete. 2024-06-15T00:48:00.062 INFO:teuthology.orchestra.run.smithi105.stdout:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-06-15T00:48:00.062 INFO:teuthology.orchestra.run.smithi105.stdout:Adding key to root@localhost authorized_keys... 2024-06-15T00:48:00.062 INFO:teuthology.orchestra.run.smithi105.stdout:Adding host smithi105... 2024-06-15T00:48:01.626 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:01 smithi105 ceph-mon[28877]: from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi105", "addr": "172.21.15.105", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:01.626 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:01 smithi105 ceph-mon[28877]: Deploying cephadm binary to smithi105 2024-06-15T00:48:03.864 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Added host 'smithi105' with addr '172.21.15.105' 2024-06-15T00:48:03.864 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.407+0000 7fbc655b7640 1 Processor -- start 2024-06-15T00:48:03.864 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.407+0000 7fbc655b7640 1 -- start start 2024-06-15T00:48:03.864 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.407+0000 7fbc655b7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60106b00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.407+0000 7fbc655b7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc601024f0 con 0x7fbc60075c00 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.407+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60106b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.408+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60106b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33136/0 (socket says 172.21.15.105:33136) 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.408+0000 7fbc5effd640 1 -- 172.21.15.105:0/151513584 learned_addr learned my addr 172.21.15.105:0/151513584 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.408+0000 7fbc5effd640 1 -- 172.21.15.105:0/151513584 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc60107040 con 0x7fbc60075c00 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.408+0000 7fbc5effd640 1 --2- 172.21.15.105:0/151513584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60106b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009e50 tx=0x7fbc58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=4a96b1262c36fa17 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc5dffb640 1 -- 172.21.15.105:0/151513584 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc58031560 con 0x7fbc60075c00 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc5dffb640 1 -- 172.21.15.105:0/151513584 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbc5802d050 con 0x7fbc60075c00 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc5dffb640 1 -- 172.21.15.105:0/151513584 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc580079a0 con 0x7fbc60075c00 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 -- 172.21.15.105:0/151513584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 msgr2=0x7fbc60106b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 --2- 172.21.15.105:0/151513584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60106b00 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 crypto rx=0x7fbc58009e50 tx=0x7fbc58009840 comp rx=0 tx=0).stop 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 -- 172.21.15.105:0/151513584 shutdown_connections 2024-06-15T00:48:03.865 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 --2- 172.21.15.105:0/151513584 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60106b00 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 -- 172.21.15.105:0/151513584 >> 172.21.15.105:0/151513584 conn(0x7fbc600fc150 msgr2=0x7fbc600fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 -- 172.21.15.105:0/151513584 shutdown_connections 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.409+0000 7fbc655b7640 1 -- 172.21.15.105:0/151513584 wait complete. 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc655b7640 1 Processor -- start 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc655b7640 1 -- start start 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc655b7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60112a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc655b7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc60112fd0 con 0x7fbc60075c00 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60112a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc5effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60112a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33148/0 (socket says 172.21.15.105:33148) 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.410+0000 7fbc5effd640 1 -- 172.21.15.105:0/2027296697 learned_addr learned my addr 172.21.15.105:0/2027296697 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc5effd640 1 -- 172.21.15.105:0/2027296697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc58009ae0 con 0x7fbc60075c00 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc5effd640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60112a90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbc580076c0 tx=0x7fbc58009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:03.866 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc58031bc0 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc601131d0 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fbc58031d20 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc580453b0 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.411+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc60113640 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.412+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fbc58038050 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.412+0000 7fbc577fe640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbc4c034810 0x7fbc4c036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.412+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fbc58066e10 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.412+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc60113640 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.412+0000 7fbc5e7fc640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbc4c034810 0x7fbc4c036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.412+0000 7fbc5e7fc640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbc4c034810 0x7fbc4c036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc48006fd0 tx=0x7fbc48006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.416+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbc60113640 con 0x7fbc60075c00 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:00.585+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi105", "addr": "172.21.15.105", "target": ["mon-mgr", ""]}) v1 -- 0x7fbc60113640 con 0x7fbc4c034810 2024-06-15T00:48:03.867 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.291+0000 7fbc577fe640 1 -- 172.21.15.105:0/2027296697 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7fbc60113640 con 0x7fbc4c034810 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbc4c034810 msgr2=0x7fbc4c036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbc4c034810 0x7fbc4c036cc0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fbc48006fd0 tx=0x7fbc48006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 msgr2=0x7fbc60112a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60112a90 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 crypto rx=0x7fbc580076c0 tx=0x7fbc58009840 comp rx=0 tx=0).stop 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 shutdown_connections 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fbc4c034810 0x7fbc4c036cc0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 --2- 172.21.15.105:0/2027296697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbc60075c00 0x7fbc60112a90 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 >> 172.21.15.105:0/2027296697 conn(0x7fbc600fc150 msgr2=0x7fbc600fcdb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 shutdown_connections 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:03.294+0000 7fbc655b7640 1 -- 172.21.15.105:0/2027296697 wait complete. 2024-06-15T00:48:03.868 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying mon service with default placement... 2024-06-15T00:48:04.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:04 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:04.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:04 smithi105 ceph-mon[28877]: Added host smithi105 2024-06-15T00:48:04.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:04 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:48:05.173 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled mon update... 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.220+0000 7f2e1fef2640 1 Processor -- start 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.220+0000 7f2e1fef2640 1 -- start start 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.221+0000 7f2e1fef2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18106b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.221+0000 7f2e1fef2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18102510 con 0x7f2e18075be0 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.221+0000 7f2e1dc8e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18106b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.221+0000 7f2e1dc8e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18106b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34530/0 (socket says 172.21.15.105:34530) 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.221+0000 7f2e1dc8e640 1 -- 172.21.15.105:0/3108039327 learned_addr learned my addr 172.21.15.105:0/3108039327 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.221+0000 7f2e1dc8e640 1 -- 172.21.15.105:0/3108039327 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18107060 con 0x7f2e18075be0 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.222+0000 7f2e1dc8e640 1 --2- 172.21.15.105:0/3108039327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18106b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009e50 tx=0x7f2e08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=f4ede603897c14e2 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:05.174 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.222+0000 7f2e1cc8c640 1 -- 172.21.15.105:0/3108039327 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e08031560 con 0x7f2e18075be0 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.222+0000 7f2e1cc8c640 1 -- 172.21.15.105:0/3108039327 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2e0802d050 con 0x7f2e18075be0 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.222+0000 7f2e1cc8c640 1 -- 172.21.15.105:0/3108039327 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e080079a0 con 0x7f2e18075be0 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- 172.21.15.105:0/3108039327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 msgr2=0x7f2e18106b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 --2- 172.21.15.105:0/3108039327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18106b20 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7f2e08009e50 tx=0x7f2e08009840 comp rx=0 tx=0).stop 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- 172.21.15.105:0/3108039327 shutdown_connections 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 --2- 172.21.15.105:0/3108039327 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18106b20 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- 172.21.15.105:0/3108039327 >> 172.21.15.105:0/3108039327 conn(0x7f2e180fc150 msgr2=0x7f2e180fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- 172.21.15.105:0/3108039327 shutdown_connections 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- 172.21.15.105:0/3108039327 wait complete. 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 Processor -- start 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- start start 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18112ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:05.175 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1fef2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e18113000 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1dc8e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18112ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1dc8e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18112ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34546/0 (socket says 172.21.15.105:34546) 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.223+0000 7f2e1dc8e640 1 -- 172.21.15.105:0/1730556716 learned_addr learned my addr 172.21.15.105:0/1730556716 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e1dc8e640 1 -- 172.21.15.105:0/1730556716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e08009ae0 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e1dc8e640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18112ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2e08007030 tx=0x7f2e08009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e08031bc0 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e18113200 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f2e08031d20 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e080453b0 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.224+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e18113670 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.225+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f2e08038050 con 0x7f2e18075be0 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.225+0000 7f2e167fc640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f2e00034810 0x7f2e00036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:05.176 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.225+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e18113670 con 0x7f2e18075be0 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.225+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f2e08066d70 con 0x7f2e18075be0 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.225+0000 7f2e1d48d640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f2e00034810 0x7f2e00036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.226+0000 7f2e1d48d640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f2e00034810 0x7f2e00036cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c006fd0 tx=0x7f2e0c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.229+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2e18113670 con 0x7f2e18075be0 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.422+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f2e18113670 con 0x7f2e00034810 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.429+0000 7f2e167fc640 1 -- 172.21.15.105:0/1730556716 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f2e18113670 con 0x7f2e00034810 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f2e00034810 msgr2=0x7f2e00036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f2e00034810 0x7f2e00036cc0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7f2e0c006fd0 tx=0x7f2e0c006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 msgr2=0x7f2e18112ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:05.177 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18112ac0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 crypto rx=0x7f2e08007030 tx=0x7f2e08009840 comp rx=0 tx=0).stop 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 shutdown_connections 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f2e00034810 0x7f2e00036cc0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 --2- 172.21.15.105:0/1730556716 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2e18075be0 0x7f2e18112ac0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 >> 172.21.15.105:0/1730556716 conn(0x7f2e180fc150 msgr2=0x7f2e180fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 shutdown_connections 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:04.431+0000 7f2e1fef2640 1 -- 172.21.15.105:0/1730556716 wait complete. 2024-06-15T00:48:05.178 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying mgr service with default placement... 2024-06-15T00:48:05.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:05 smithi105 ceph-mon[28877]: from='client.14140 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:05.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:05 smithi105 ceph-mon[28877]: Saving service mon spec with placement count:5 2024-06-15T00:48:05.720 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:05 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:06.341 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled mgr update... 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.603+0000 7fa5802cc640 1 Processor -- start 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.603+0000 7fa5802cc640 1 -- start start 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.603+0000 7fa5802cc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.603+0000 7fa5802cc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa578104c00 con 0x7fa578075c40 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.604+0000 7fa57e068640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.604+0000 7fa57e068640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34550/0 (socket says 172.21.15.105:34550) 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.604+0000 7fa57e068640 1 -- 172.21.15.105:0/341307058 learned_addr learned my addr 172.21.15.105:0/341307058 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.604+0000 7fa57e068640 1 -- 172.21.15.105:0/341307058 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa578104d40 con 0x7fa578075c40 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa57e068640 1 --2- 172.21.15.105:0/341307058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781046c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa574009840 tx=0x7fa5740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2d067127cfc4df0c server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa57d066640 1 -- 172.21.15.105:0/341307058 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa574032440 con 0x7fa578075c40 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa57d066640 1 -- 172.21.15.105:0/341307058 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa574032a10 con 0x7fa578075c40 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa57d066640 1 -- 172.21.15.105:0/341307058 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa574031a00 con 0x7fa578075c40 2024-06-15T00:48:06.342 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 -- 172.21.15.105:0/341307058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 msgr2=0x7fa5781046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 --2- 172.21.15.105:0/341307058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781046c0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 crypto rx=0x7fa574009840 tx=0x7fa5740091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 -- 172.21.15.105:0/341307058 shutdown_connections 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 --2- 172.21.15.105:0/341307058 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781046c0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 -- 172.21.15.105:0/341307058 >> 172.21.15.105:0/341307058 conn(0x7fa5780fc150 msgr2=0x7fa5780fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 -- 172.21.15.105:0/341307058 shutdown_connections 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.605+0000 7fa5802cc640 1 -- 172.21.15.105:0/341307058 wait complete. 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa5802cc640 1 Processor -- start 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa5802cc640 1 -- start start 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa5802cc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781108c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa5802cc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa578110e00 con 0x7fa578075c40 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa57e068640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781108c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa57e068640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781108c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34554/0 (socket says 172.21.15.105:34554) 2024-06-15T00:48:06.343 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa57e068640 1 -- 172.21.15.105:0/4233711772 learned_addr learned my addr 172.21.15.105:0/4233711772 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.606+0000 7fa57e068640 1 -- 172.21.15.105:0/4233711772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa574009450 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.607+0000 7fa57e068640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781108c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa574006f00 tx=0x7fa574008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.607+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa574031e20 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.607+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa578111000 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.607+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7fa57403d8c0 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.607+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa57403b840 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.607+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa578111470 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.608+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fa574039070 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.608+0000 7fa56affd640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fa564034810 0x7fa564036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.608+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fa574067670 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.608+0000 7fa57d867640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fa564034810 0x7fa564036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.608+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa57805ead0 con 0x7fa578075c40 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.608+0000 7fa57d867640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fa564034810 0x7fa564036cc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa56c006fd0 tx=0x7fa56c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:06.344 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.613+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa574041070 con 0x7fa578075c40 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.790+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa5780fd5b0 con 0x7fa564034810 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.796+0000 7fa56affd640 1 -- 172.21.15.105:0/4233711772 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa5780fd5b0 con 0x7fa564034810 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fa564034810 msgr2=0x7fa564036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fa564034810 0x7fa564036cc0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7fa56c006fd0 tx=0x7fa56c006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 msgr2=0x7fa5781108c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781108c0 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 crypto rx=0x7fa574006f00 tx=0x7fa574008040 comp rx=0 tx=0).stop 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 shutdown_connections 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fa564034810 0x7fa564036cc0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 --2- 172.21.15.105:0/4233711772 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa578075c40 0x7fa5781108c0 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 >> 172.21.15.105:0/4233711772 conn(0x7fa5780fc150 msgr2=0x7fa5780fcde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 shutdown_connections 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:05.798+0000 7fa5802cc640 1 -- 172.21.15.105:0/4233711772 wait complete. 2024-06-15T00:48:06.345 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying crash service with default placement... 2024-06-15T00:48:06.953 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:06 smithi105 ceph-mon[28877]: from='client.14142 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:06.954 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:06 smithi105 ceph-mon[28877]: Saving service mgr spec with placement count:2 2024-06-15T00:48:06.954 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:06 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:06.954 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:06 smithi105 ceph-mon[28877]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:08.094 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:07 smithi105 ceph-mon[28877]: from='client.14144 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:08.095 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:07 smithi105 ceph-mon[28877]: Saving service crash spec with placement * 2024-06-15T00:48:08.095 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:07 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:08.095 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:07 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:08.102 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled crash update... 2024-06-15T00:48:08.102 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.770+0000 7f163fc87640 1 Processor -- start 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163fc87640 1 -- start start 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163fc87640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638104700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163fc87640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1638104c40 con 0x7f1638075d00 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163da23640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638104700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163da23640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638104700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34556/0 (socket says 172.21.15.105:34556) 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163da23640 1 -- 172.21.15.105:0/2998363829 learned_addr learned my addr 172.21.15.105:0/2998363829 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.771+0000 7f163da23640 1 -- 172.21.15.105:0/2998363829 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1638104d80 con 0x7f1638075d00 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.772+0000 7f163da23640 1 --2- 172.21.15.105:0/2998363829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638104700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1634009840 tx=0x7f16340091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=b6171943f49de1a0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.772+0000 7f163ca21640 1 -- 172.21.15.105:0/2998363829 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1634032440 con 0x7f1638075d00 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.772+0000 7f163ca21640 1 -- 172.21.15.105:0/2998363829 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f1634032a10 con 0x7f1638075d00 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.772+0000 7f163ca21640 1 -- 172.21.15.105:0/2998363829 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1634031a00 con 0x7f1638075d00 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 -- 172.21.15.105:0/2998363829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 msgr2=0x7f1638104700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:08.103 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 --2- 172.21.15.105:0/2998363829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638104700 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 crypto rx=0x7f1634009840 tx=0x7f16340091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 -- 172.21.15.105:0/2998363829 shutdown_connections 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 --2- 172.21.15.105:0/2998363829 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638104700 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 -- 172.21.15.105:0/2998363829 >> 172.21.15.105:0/2998363829 conn(0x7f16380fc190 msgr2=0x7f16380fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 -- 172.21.15.105:0/2998363829 shutdown_connections 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.773+0000 7f163fc87640 1 -- 172.21.15.105:0/2998363829 wait complete. 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163fc87640 1 Processor -- start 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163fc87640 1 -- start start 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163fc87640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638102030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163fc87640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1638102570 con 0x7f1638075d00 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163da23640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638102030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163da23640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638102030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34570/0 (socket says 172.21.15.105:34570) 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163da23640 1 -- 172.21.15.105:0/437083992 learned_addr learned my addr 172.21.15.105:0/437083992 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163da23640 1 -- 172.21.15.105:0/437083992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1634009450 con 0x7f1638075d00 2024-06-15T00:48:08.104 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.774+0000 7f163da23640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638102030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16340316e0 tx=0x7f1634008040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.775+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1634031e20 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.775+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1009+0+0 (secure 0 0 0) 0x7f163403d8c0 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.775+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1638100600 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.775+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f163403b7a0 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.775+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1638100a70 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.775+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f1634039070 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.776+0000 7f162e7fc640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f1624034810 0x7f1624036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.776+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f1634067660 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.776+0000 7f163d222640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f1624034810 0x7f1624036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.776+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f163805ead0 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.776+0000 7f163d222640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f1624034810 0x7f1624036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1628006fd0 tx=0x7f1628006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.780+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1634041070 con 0x7f1638075d00 2024-06-15T00:48:08.105 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.960+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f1638101030 con 0x7f1624034810 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.964+0000 7f162e7fc640 1 -- 172.21.15.105:0/437083992 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f1638101030 con 0x7f1624034810 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f1624034810 msgr2=0x7f1624036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f1624034810 0x7f1624036cc0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f1628006fd0 tx=0x7f1628006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 msgr2=0x7f1638102030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638102030 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 crypto rx=0x7f16340316e0 tx=0x7f1634008040 comp rx=0 tx=0).stop 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 shutdown_connections 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f1624034810 0x7f1624036cc0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 --2- 172.21.15.105:0/437083992 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1638075d00 0x7f1638102030 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 >> 172.21.15.105:0/437083992 conn(0x7f16380fc190 msgr2=0x7f16380fcd90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 shutdown_connections 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:06.966+0000 7f163fc87640 1 -- 172.21.15.105:0/437083992 wait complete. 2024-06-15T00:48:08.106 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying prometheus service with default placement... 2024-06-15T00:48:09.461 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:09 smithi105 ceph-mon[28877]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:09.462 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:09 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:09.462 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:09 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:09.468 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled prometheus update... 2024-06-15T00:48:09.468 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.581+0000 7f0551bd9640 1 Processor -- start 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.581+0000 7f0551bd9640 1 -- start start 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.582+0000 7f0551bd9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c101e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.582+0000 7f0551bd9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1023d0 con 0x7f054c101a80 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.582+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.582+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c101e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34578/0 (socket says 172.21.15.105:34578) 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.582+0000 7f054b7fe640 1 -- 172.21.15.105:0/3950523402 learned_addr learned my addr 172.21.15.105:0/3950523402 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.582+0000 7f054b7fe640 1 -- 172.21.15.105:0/3950523402 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c102510 con 0x7f054c101a80 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.583+0000 7f054b7fe640 1 --2- 172.21.15.105:0/3950523402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c101e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f053c009840 tx=0x7f053c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cb669223c871c9e3 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.583+0000 7f054a7fc640 1 -- 172.21.15.105:0/3950523402 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c032440 con 0x7f054c101a80 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.583+0000 7f054a7fc640 1 -- 172.21.15.105:0/3950523402 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f053c032a50 con 0x7f054c101a80 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.583+0000 7f054a7fc640 1 -- 172.21.15.105:0/3950523402 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c031aa0 con 0x7f054c101a80 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 -- 172.21.15.105:0/3950523402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 msgr2=0x7f054c101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:09.469 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 --2- 172.21.15.105:0/3950523402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c101e90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f053c009840 tx=0x7f053c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 -- 172.21.15.105:0/3950523402 shutdown_connections 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 --2- 172.21.15.105:0/3950523402 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c101e90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 -- 172.21.15.105:0/3950523402 >> 172.21.15.105:0/3950523402 conn(0x7f054c0fd2e0 msgr2=0x7f054c0ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 -- 172.21.15.105:0/3950523402 shutdown_connections 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 -- 172.21.15.105:0/3950523402 wait complete. 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 Processor -- start 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.584+0000 7f0551bd9640 1 -- start start 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0551bd9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c1107f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0551bd9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c110d30 con 0x7f054c101a80 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c1107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f054b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c1107f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34590/0 (socket says 172.21.15.105:34590) 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f054b7fe640 1 -- 172.21.15.105:0/3737508042 learned_addr learned my addr 172.21.15.105:0/3737508042 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:09.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f054b7fe640 1 -- 172.21.15.105:0/3737508042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f053c009450 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f054b7fe640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c1107f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f053c031780 tx=0x7f053c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c03b440 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c110f30 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f053c03ba50 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f053c03aab0 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.585+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c111340 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.586+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f053c039070 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.586+0000 7f0533fff640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f0534034810 0x7f0534036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.586+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c04b480 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.586+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f053c067260 con 0x7f054c101a80 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.586+0000 7f054affd640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f0534034810 0x7f0534036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.587+0000 7f054affd640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f0534034810 0x7f0534036cc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0540006fd0 tx=0x7f0540006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:09.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.591+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f053c037070 con 0x7f054c101a80 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.783+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f054c0fe4d0 con 0x7f0534034810 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.787+0000 7f0533fff640 1 -- 172.21.15.105:0/3737508042 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f054c0fe4d0 con 0x7f0534034810 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f0534034810 msgr2=0x7f0534036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f0534034810 0x7f0534036cc0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 crypto rx=0x7f0540006fd0 tx=0x7f0540006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 msgr2=0x7f054c1107f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c1107f0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 crypto rx=0x7f053c031780 tx=0x7f053c02c040 comp rx=0 tx=0).stop 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 shutdown_connections 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f0534034810 0x7f0534036cc0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 --2- 172.21.15.105:0/3737508042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f054c101a80 0x7f054c1107f0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 >> 172.21.15.105:0/3737508042 conn(0x7f054c0fd2e0 msgr2=0x7f054c0fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 shutdown_connections 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:08.789+0000 7f0551bd9640 1 -- 172.21.15.105:0/3737508042 wait complete. 2024-06-15T00:48:09.472 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying grafana service with default placement... 2024-06-15T00:48:10.350 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:10 smithi105 ceph-mon[28877]: from='client.14146 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:10.350 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:10 smithi105 ceph-mon[28877]: Saving service prometheus spec with placement count:1 2024-06-15T00:48:10.350 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:10 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:11.195 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled grafana update... 2024-06-15T00:48:11.195 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.357+0000 7fd59ea61640 1 Processor -- start 2024-06-15T00:48:11.195 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.357+0000 7fd59ea61640 1 -- start start 2024-06-15T00:48:11.195 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.358+0000 7fd59ea61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd598101e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.358+0000 7fd59ea61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5981023d0 con 0x7fd598101a80 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.358+0000 7fd597fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd598101e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.358+0000 7fd597fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd598101e90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34592/0 (socket says 172.21.15.105:34592) 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.358+0000 7fd597fff640 1 -- 172.21.15.105:0/3957153229 learned_addr learned my addr 172.21.15.105:0/3957153229 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.358+0000 7fd597fff640 1 -- 172.21.15.105:0/3957153229 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd598102510 con 0x7fd598101a80 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.359+0000 7fd597fff640 1 --2- 172.21.15.105:0/3957153229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd598101e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd588009840 tx=0x7fd5880091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ce4f7fd551689854 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.359+0000 7fd596ffd640 1 -- 172.21.15.105:0/3957153229 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd588032440 con 0x7fd598101a80 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.359+0000 7fd596ffd640 1 -- 172.21.15.105:0/3957153229 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd588032a50 con 0x7fd598101a80 2024-06-15T00:48:11.196 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.359+0000 7fd596ffd640 1 -- 172.21.15.105:0/3957153229 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd588031aa0 con 0x7fd598101a80 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 -- 172.21.15.105:0/3957153229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 msgr2=0x7fd598101e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 --2- 172.21.15.105:0/3957153229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd598101e90 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 crypto rx=0x7fd588009840 tx=0x7fd5880091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 -- 172.21.15.105:0/3957153229 shutdown_connections 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 --2- 172.21.15.105:0/3957153229 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd598101e90 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 -- 172.21.15.105:0/3957153229 >> 172.21.15.105:0/3957153229 conn(0x7fd5980fd2e0 msgr2=0x7fd5980ff6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 -- 172.21.15.105:0/3957153229 shutdown_connections 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.360+0000 7fd59ea61640 1 -- 172.21.15.105:0/3957153229 wait complete. 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd59ea61640 1 Processor -- start 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd59ea61640 1 -- start start 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd59ea61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd5981107f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd59ea61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd598110d30 con 0x7fd598101a80 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd597fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd5981107f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:11.197 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd597fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd5981107f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34598/0 (socket says 172.21.15.105:34598) 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd597fff640 1 -- 172.21.15.105:0/1253835393 learned_addr learned my addr 172.21.15.105:0/1253835393 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.361+0000 7fd597fff640 1 -- 172.21.15.105:0/1253835393 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd588009450 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.362+0000 7fd597fff640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd5981107f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd588031780 tx=0x7fd58802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.362+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58803b440 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.362+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd58803ba50 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.362+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd598110f30 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.362+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd58803aab0 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.362+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd598111340 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.363+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fd588039070 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.363+0000 7fd594ff9640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fd580034810 0x7fd580036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.363+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59804b480 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.363+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fd58803b5a0 con 0x7fd598101a80 2024-06-15T00:48:11.198 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.363+0000 7fd5977fe640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fd580034810 0x7fd580036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.364+0000 7fd5977fe640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fd580034810 0x7fd580036cc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd58c006fd0 tx=0x7fd58c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.368+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd588030930 con 0x7fd598101a80 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.549+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7fd5980fe4d0 con 0x7fd580034810 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.553+0000 7fd594ff9640 1 -- 172.21.15.105:0/1253835393 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7fd5980fe4d0 con 0x7fd580034810 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fd580034810 msgr2=0x7fd580036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fd580034810 0x7fd580036cc0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7fd58c006fd0 tx=0x7fd58c006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 msgr2=0x7fd5981107f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd5981107f0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 crypto rx=0x7fd588031780 tx=0x7fd58802c040 comp rx=0 tx=0).stop 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 shutdown_connections 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fd580034810 0x7fd580036cc0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 --2- 172.21.15.105:0/1253835393 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd598101a80 0x7fd5981107f0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 >> 172.21.15.105:0/1253835393 conn(0x7fd5980fd2e0 msgr2=0x7fd5980fddc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 shutdown_connections 2024-06-15T00:48:11.199 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:10.556+0000 7fd59ea61640 1 -- 172.21.15.105:0/1253835393 wait complete. 2024-06-15T00:48:11.200 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying node-exporter service with default placement... 2024-06-15T00:48:11.582 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:11 smithi105 ceph-mon[28877]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:11.582 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:11 smithi105 ceph-mon[28877]: from='client.14148 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:11.583 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:11 smithi105 ceph-mon[28877]: Saving service grafana spec with placement count:1 2024-06-15T00:48:11.583 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:11 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:11.583 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:11 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:13.034 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:12 smithi105 ceph-mon[28877]: from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:13.035 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:12 smithi105 ceph-mon[28877]: Saving service node-exporter spec with placement * 2024-06-15T00:48:13.035 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:12 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:13.035 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:12 smithi105 ceph-mon[28877]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:13.233 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled node-exporter update... 2024-06-15T00:48:13.233 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.583+0000 7f3350def640 1 Processor -- start 2024-06-15T00:48:13.233 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.583+0000 7f3350def640 1 -- start start 2024-06-15T00:48:13.233 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.584+0000 7f3350def640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c100250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.584+0000 7f3350def640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c100790 con 0x7f334c0ffe40 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.584+0000 7f334a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c100250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.584+0000 7f334a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c100250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34610/0 (socket says 172.21.15.105:34610) 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.584+0000 7f334a59c640 1 -- 172.21.15.105:0/3267848609 learned_addr learned my addr 172.21.15.105:0/3267848609 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.584+0000 7f334a59c640 1 -- 172.21.15.105:0/3267848609 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f334c1008d0 con 0x7f334c0ffe40 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.585+0000 7f334a59c640 1 --2- 172.21.15.105:0/3267848609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c100250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f333c009840 tx=0x7f333c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=94e8178d4f269bb0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.585+0000 7f3349d9b640 1 -- 172.21.15.105:0/3267848609 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f333c032440 con 0x7f334c0ffe40 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.585+0000 7f3349d9b640 1 -- 172.21.15.105:0/3267848609 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f333c032a50 con 0x7f334c0ffe40 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.585+0000 7f3349d9b640 1 -- 172.21.15.105:0/3267848609 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f333c031aa0 con 0x7f334c0ffe40 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.585+0000 7f3350def640 1 -- 172.21.15.105:0/3267848609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 msgr2=0x7f334c100250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.585+0000 7f3350def640 1 --2- 172.21.15.105:0/3267848609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c100250 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 crypto rx=0x7f333c009840 tx=0x7f333c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 -- 172.21.15.105:0/3267848609 shutdown_connections 2024-06-15T00:48:13.234 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 --2- 172.21.15.105:0/3267848609 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c100250 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 -- 172.21.15.105:0/3267848609 >> 172.21.15.105:0/3267848609 conn(0x7f334c0fb8c0 msgr2=0x7f334c0fdcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 -- 172.21.15.105:0/3267848609 shutdown_connections 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 -- 172.21.15.105:0/3267848609 wait complete. 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 Processor -- start 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 -- start start 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c110a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f3350def640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f334c110fa0 con 0x7f334c0ffe40 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f334a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c110a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f334a59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c110a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34626/0 (socket says 172.21.15.105:34626) 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.586+0000 7f334a59c640 1 -- 172.21.15.105:0/1238723372 learned_addr learned my addr 172.21.15.105:0/1238723372 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f334a59c640 1 -- 172.21.15.105:0/1238723372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f333c009450 con 0x7f334c0ffe40 2024-06-15T00:48:13.235 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f334a59c640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c110a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f333c009420 tx=0x7f333c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f333c03b440 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f333c03ba50 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f334c1111a0 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f333c03a9e0 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.587+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f334c111610 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.588+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7f333c039070 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.588+0000 7f3342ffd640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3338034860 0x7f3338036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.588+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7f333c066f90 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.588+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f334c05ead0 con 0x7f334c0ffe40 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.588+0000 7f3343fff640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3338034860 0x7f3338036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.588+0000 7f3343fff640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3338034860 0x7f3338036d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3334006fd0 tx=0x7f3334006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:13.236 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.592+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f333c037070 con 0x7f334c0ffe40 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.775+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f334c0fcad0 con 0x7f3338034860 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.779+0000 7f3342ffd640 1 -- 172.21.15.105:0/1238723372 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f334c0fcad0 con 0x7f3338034860 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3338034860 msgr2=0x7f3338036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3338034860 0x7f3338036d10 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f3334006fd0 tx=0x7f3334006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 msgr2=0x7f334c110a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c110a60 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7f333c009420 tx=0x7f333c02c040 comp rx=0 tx=0).stop 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 shutdown_connections 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7f3338034860 0x7f3338036d10 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 --2- 172.21.15.105:0/1238723372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f334c0ffe40 0x7f334c110a60 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.781+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 >> 172.21.15.105:0/1238723372 conn(0x7f334c0fb8c0 msgr2=0x7f334c0fc3c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.782+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 shutdown_connections 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:11.782+0000 7f3350def640 1 -- 172.21.15.105:0/1238723372 wait complete. 2024-06-15T00:48:13.237 INFO:teuthology.orchestra.run.smithi105.stdout:Deploying alertmanager service with default placement... 2024-06-15T00:48:14.364 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Scheduled alertmanager update... 2024-06-15T00:48:14.364 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.601+0000 7fc37ad09640 1 Processor -- start 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc37ad09640 1 -- start start 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc37ad09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374101430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc37ad09640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374101970 con 0x7fc374101020 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc378aa5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374101430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc378aa5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374101430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43410/0 (socket says 172.21.15.105:43410) 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc378aa5640 1 -- 172.21.15.105:0/3072184924 learned_addr learned my addr 172.21.15.105:0/3072184924 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.602+0000 7fc378aa5640 1 -- 172.21.15.105:0/3072184924 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374101ab0 con 0x7fc374101020 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.603+0000 7fc378aa5640 1 --2- 172.21.15.105:0/3072184924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374101430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009840 tx=0x7fc36c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ae388427c7f96b76 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.603+0000 7fc3737fe640 1 -- 172.21.15.105:0/3072184924 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c032440 con 0x7fc374101020 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.603+0000 7fc3737fe640 1 -- 172.21.15.105:0/3072184924 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc36c032a50 con 0x7fc374101020 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.603+0000 7fc3737fe640 1 -- 172.21.15.105:0/3072184924 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c031aa0 con 0x7fc374101020 2024-06-15T00:48:14.365 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 -- 172.21.15.105:0/3072184924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 msgr2=0x7fc374101430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 --2- 172.21.15.105:0/3072184924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374101430 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 crypto rx=0x7fc36c009840 tx=0x7fc36c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 -- 172.21.15.105:0/3072184924 shutdown_connections 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 --2- 172.21.15.105:0/3072184924 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374101430 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 -- 172.21.15.105:0/3072184924 >> 172.21.15.105:0/3072184924 conn(0x7fc3740fb990 msgr2=0x7fc3740fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 -- 172.21.15.105:0/3072184924 shutdown_connections 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 -- 172.21.15.105:0/3072184924 wait complete. 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.604+0000 7fc37ad09640 1 Processor -- start 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc37ad09640 1 -- start start 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc37ad09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374112a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc37ad09640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc374112f90 con 0x7fc374101020 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc378aa5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374112a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc378aa5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374112a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43420/0 (socket says 172.21.15.105:43420) 2024-06-15T00:48:14.366 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc378aa5640 1 -- 172.21.15.105:0/3786591696 learned_addr learned my addr 172.21.15.105:0/3786591696 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc378aa5640 1 -- 172.21.15.105:0/3786591696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36c009450 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc378aa5640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374112a50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc36c031780 tx=0x7fc36c02c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.605+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c03b440 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.606+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc36c03ba50 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.606+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc374113190 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.606+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc36c03a960 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.606+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3741135a0 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.606+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fc36c039070 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.607+0000 7fc3717fa640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc360034800 0x7fc360036cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.607+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37404b480 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.607+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc36c0671d0 con 0x7fc374101020 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.607+0000 7fc373fff640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc360034800 0x7fc360036cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:14.367 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.607+0000 7fc373fff640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc360034800 0x7fc360036cb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc36400ad90 tx=0x7fc364009410 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:14.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.611+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc36c037020 con 0x7fc374101020 2024-06-15T00:48:14.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.782+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3740fcb10 con 0x7fc360034800 2024-06-15T00:48:14.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.786+0000 7fc3717fa640 1 -- 172.21.15.105:0/3786591696 <== mgr.14118 v2:172.21.15.105:6800/2469050433 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fc3740fcb10 con 0x7fc360034800 2024-06-15T00:48:14.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc360034800 msgr2=0x7fc360036cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:14.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc360034800 0x7fc360036cb0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7fc36400ad90 tx=0x7fc364009410 comp rx=0 tx=0).stop 2024-06-15T00:48:14.368 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 msgr2=0x7fc374112a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374112a50 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 crypto rx=0x7fc36c031780 tx=0x7fc36c02c040 comp rx=0 tx=0).stop 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 shutdown_connections 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc360034800 0x7fc360036cb0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 --2- 172.21.15.105:0/3786591696 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc374101020 0x7fc374112a50 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 >> 172.21.15.105:0/3786591696 conn(0x7fc3740fb990 msgr2=0x7fc3740fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 shutdown_connections 2024-06-15T00:48:14.369 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:13.788+0000 7fc37ad09640 1 -- 172.21.15.105:0/3786591696 wait complete. 2024-06-15T00:48:15.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:14 smithi105 ceph-mon[28877]: from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:15.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:14 smithi105 ceph-mon[28877]: Saving service alertmanager spec with placement count:1 2024-06-15T00:48:15.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:14 smithi105 ceph-mon[28877]: from='mgr.14118 172.21.15.105:0/1713029507' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:15.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:14 smithi105 ceph-mon[28877]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.762+0000 7fb489be9640 1 Processor -- start 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.762+0000 7fb489be9640 1 -- start start 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.762+0000 7fb489be9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484102360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.762+0000 7fb489be9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4841028a0 con 0x7fb484101f50 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.763+0000 7fb4837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484102360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.763+0000 7fb4837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484102360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43424/0 (socket says 172.21.15.105:43424) 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.763+0000 7fb4837fe640 1 -- 172.21.15.105:0/2393898083 learned_addr learned my addr 172.21.15.105:0/2393898083 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.763+0000 7fb4837fe640 1 -- 172.21.15.105:0/2393898083 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4841029e0 con 0x7fb484101f50 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb4837fe640 1 --2- 172.21.15.105:0/2393898083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484102360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb474009840 tx=0x7fb4740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=fc86662c45262387 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:15.504 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb482ffd640 1 -- 172.21.15.105:0/2393898083 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb474032440 con 0x7fb484101f50 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb482ffd640 1 -- 172.21.15.105:0/2393898083 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb474032a50 con 0x7fb484101f50 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb482ffd640 1 -- 172.21.15.105:0/2393898083 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb474031aa0 con 0x7fb484101f50 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 -- 172.21.15.105:0/2393898083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 msgr2=0x7fb484102360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 --2- 172.21.15.105:0/2393898083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484102360 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 crypto rx=0x7fb474009840 tx=0x7fb4740091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 -- 172.21.15.105:0/2393898083 shutdown_connections 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 --2- 172.21.15.105:0/2393898083 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484102360 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 -- 172.21.15.105:0/2393898083 >> 172.21.15.105:0/2393898083 conn(0x7fb4840fb8b0 msgr2=0x7fb4840fdce0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 -- 172.21.15.105:0/2393898083 shutdown_connections 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.764+0000 7fb489be9640 1 -- 172.21.15.105:0/2393898083 wait complete. 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb489be9640 1 Processor -- start 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb489be9640 1 -- start start 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb489be9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484110850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:15.505 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb489be9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb484110d90 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb4837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484110850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb4837fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484110850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43430/0 (socket says 172.21.15.105:43430) 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb4837fe640 1 -- 172.21.15.105:0/278301760 learned_addr learned my addr 172.21.15.105:0/278301760 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.765+0000 7fb4837fe640 1 -- 172.21.15.105:0/278301760 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb474009450 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.766+0000 7fb4837fe640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484110850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb474031780 tx=0x7fb47402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.766+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb47403b440 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.766+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb47403ba50 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.766+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb47403aab0 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.766+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb484110f90 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.766+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4841113a0 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.767+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fb474039070 con 0x7fb484101f50 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.767+0000 7fb480ff9640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb470034860 0x7fb470036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:15.506 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.767+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb4740670b0 con 0x7fb484101f50 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.767+0000 7fb47bfff640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb470034860 0x7fb470036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.767+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48404b480 con 0x7fb484101f50 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.767+0000 7fb47bfff640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb470034860 0x7fb470036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb46c006fd0 tx=0x7fb46c006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.771+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb47403f080 con 0x7fb484101f50 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.938+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7fb484101750 con 0x7fb484101f50 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.944+0000 7fb480ff9640 1 -- 172.21.15.105:0/278301760 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v7)=0 v7) v1 ==== 142+0+0 (secure 0 0 0) 0x7fb474037070 con 0x7fb484101f50 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.946+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb470034860 msgr2=0x7fb470036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.946+0000 7fb489be9640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb470034860 0x7fb470036d10 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fb46c006fd0 tx=0x7fb46c006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.946+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 msgr2=0x7fb484110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.946+0000 7fb489be9640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484110850 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fb474031780 tx=0x7fb47402c040 comp rx=0 tx=0).stop 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.948+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 shutdown_connections 2024-06-15T00:48:15.507 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.948+0000 7fb489be9640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb470034860 0x7fb470036d10 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:15.508 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.948+0000 7fb489be9640 1 --2- 172.21.15.105:0/278301760 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb484101f50 0x7fb484110850 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:15.508 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.948+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 >> 172.21.15.105:0/278301760 conn(0x7fb4840fb8b0 msgr2=0x7fb4840fc730 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:15.508 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.948+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 shutdown_connections 2024-06-15T00:48:15.508 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:14.948+0000 7fb489be9640 1 -- 172.21.15.105:0/278301760 wait complete. 2024-06-15T00:48:16.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:15 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/278301760' entity='client.admin' 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbdd27f640 1 Processor -- start 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbdd27f640 1 -- start start 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbdd27f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd81002b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbdd27f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd81007f0 con 0x7fdbd80ffea0 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbd6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd81002b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbd6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd81002b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43442/0 (socket says 172.21.15.105:43442) 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbd6ffd640 1 -- 172.21.15.105:0/644205597 learned_addr learned my addr 172.21.15.105:0/644205597 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.885+0000 7fdbd6ffd640 1 -- 172.21.15.105:0/644205597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd8100930 con 0x7fdbd80ffea0 2024-06-15T00:48:16.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.886+0000 7fdbd6ffd640 1 --2- 172.21.15.105:0/644205597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd81002b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009840 tx=0x7fdbc80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=2f1872981fb59b8b server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.886+0000 7fdbd5ffb640 1 -- 172.21.15.105:0/644205597 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc8032440 con 0x7fdbd80ffea0 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.886+0000 7fdbd5ffb640 1 -- 172.21.15.105:0/644205597 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbc8032a50 con 0x7fdbd80ffea0 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.886+0000 7fdbd5ffb640 1 -- 172.21.15.105:0/644205597 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc8031aa0 con 0x7fdbd80ffea0 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 -- 172.21.15.105:0/644205597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 msgr2=0x7fdbd81002b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 --2- 172.21.15.105:0/644205597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd81002b0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009840 tx=0x7fdbc80091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 -- 172.21.15.105:0/644205597 shutdown_connections 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 --2- 172.21.15.105:0/644205597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd81002b0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 -- 172.21.15.105:0/644205597 >> 172.21.15.105:0/644205597 conn(0x7fdbd80fb960 msgr2=0x7fdbd80fdd70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 -- 172.21.15.105:0/644205597 shutdown_connections 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.887+0000 7fdbdd27f640 1 -- 172.21.15.105:0/644205597 wait complete. 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbdd27f640 1 Processor -- start 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbdd27f640 1 -- start start 2024-06-15T00:48:16.612 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbdd27f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd8110860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbdd27f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbd8110da0 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbd6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd8110860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbd6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd8110860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43446/0 (socket says 172.21.15.105:43446) 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbd6ffd640 1 -- 172.21.15.105:0/4134815286 learned_addr learned my addr 172.21.15.105:0/4134815286 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbd6ffd640 1 -- 172.21.15.105:0/4134815286 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbc8009450 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbd6ffd640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd8110860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009420 tx=0x7fdbc802c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc803b440 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbc803ba50 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbd8110fa0 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.888+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbc803aa20 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.889+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbd8111410 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.889+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fdbc8039070 con 0x7fdbd80ffea0 2024-06-15T00:48:16.613 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.889+0000 7fdbbf7fe640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fdbc0034860 0x7fdbc0036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.889+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fdbc8066e20 con 0x7fdbd80ffea0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.889+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbd805ead0 con 0x7fdbd80ffea0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.890+0000 7fdbd67fc640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fdbc0034860 0x7fdbc0036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.890+0000 7fdbd67fc640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fdbc0034860 0x7fdbc0036d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc006fd0 tx=0x7fdbcc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:15.894+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdbc803f080 con 0x7fdbd80ffea0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.062+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fdbd804b480 con 0x7fdbd80ffea0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.064+0000 7fdbbf7fe640 1 -- 172.21.15.105:0/4134815286 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v8)=0 v8) v1 ==== 130+0+0 (secure 0 0 0) 0x7fdbc80650c0 con 0x7fdbd80ffea0 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fdbc0034860 msgr2=0x7fdbc0036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fdbc0034860 0x7fdbc0036d10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc006fd0 tx=0x7fdbcc006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 msgr2=0x7fdbd8110860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd8110860 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 crypto rx=0x7fdbc8009420 tx=0x7fdbc802c040 comp rx=0 tx=0).stop 2024-06-15T00:48:16.614 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 shutdown_connections 2024-06-15T00:48:16.615 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fdbc0034860 0x7fdbc0036d10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:16.615 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 --2- 172.21.15.105:0/4134815286 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbd80ffea0 0x7fdbd8110860 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:16.615 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 >> 172.21.15.105:0/4134815286 conn(0x7fdbd80fb960 msgr2=0x7fdbd80fc4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:16.615 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 shutdown_connections 2024-06-15T00:48:16.615 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:16.067+0000 7fdbdd27f640 1 -- 172.21.15.105:0/4134815286 wait complete. 2024-06-15T00:48:16.615 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling the dashboard module... 2024-06-15T00:48:17.218 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:17 smithi105 ceph-mon[28877]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:17.219 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:17 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/4134815286' entity='client.admin' 2024-06-15T00:48:18.335 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:18 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1864471207' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-06-15T00:48:18.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.000+0000 7fc7e123f640 1 Processor -- start 2024-06-15T00:48:18.653 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.000+0000 7fc7e123f640 1 -- start start 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.000+0000 7fc7e123f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc101f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.000+0000 7fc7e123f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc1024a0 con 0x7fc7dc101b50 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.001+0000 7fc7dad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc101f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.001+0000 7fc7dad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc101f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43456/0 (socket says 172.21.15.105:43456) 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.001+0000 7fc7dad9d640 1 -- 172.21.15.105:0/3137132817 learned_addr learned my addr 172.21.15.105:0/3137132817 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.001+0000 7fc7dad9d640 1 -- 172.21.15.105:0/3137132817 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7dc1025e0 con 0x7fc7dc101b50 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.001+0000 7fc7dad9d640 1 --2- 172.21.15.105:0/3137132817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc101f60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009840 tx=0x7fc7c40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=a87609b9991b761c server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7d9d9b640 1 -- 172.21.15.105:0/3137132817 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c4032440 con 0x7fc7dc101b50 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7d9d9b640 1 -- 172.21.15.105:0/3137132817 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7c4032a50 con 0x7fc7dc101b50 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7d9d9b640 1 -- 172.21.15.105:0/3137132817 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c4031aa0 con 0x7fc7dc101b50 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 -- 172.21.15.105:0/3137132817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 msgr2=0x7fc7dc101f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:18.654 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 --2- 172.21.15.105:0/3137132817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc101f60 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4009840 tx=0x7fc7c40091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 -- 172.21.15.105:0/3137132817 shutdown_connections 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 --2- 172.21.15.105:0/3137132817 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc101f60 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 -- 172.21.15.105:0/3137132817 >> 172.21.15.105:0/3137132817 conn(0x7fc7dc0fd350 msgr2=0x7fc7dc0ff760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 -- 172.21.15.105:0/3137132817 shutdown_connections 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.002+0000 7fc7e123f640 1 -- 172.21.15.105:0/3137132817 wait complete. 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7e123f640 1 Processor -- start 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7e123f640 1 -- start start 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7e123f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc110830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7e123f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc7dc110d70 con 0x7fc7dc101b50 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7dad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc110830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7dad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc110830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43462/0 (socket says 172.21.15.105:43462) 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7dad9d640 1 -- 172.21.15.105:0/1864471207 learned_addr learned my addr 172.21.15.105:0/1864471207 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.003+0000 7fc7dad9d640 1 -- 172.21.15.105:0/1864471207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc7c4009450 con 0x7fc7dc101b50 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.004+0000 7fc7dad9d640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc110830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4031780 tx=0x7fc7c402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:18.655 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.004+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c403b440 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.004+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc7dc110f70 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.004+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc7c403ba50 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.004+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc7c403aab0 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.004+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc7dc111380 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.005+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 8) v1 ==== 41946+0+0 (secure 0 0 0) 0x7fc7c4039070 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.005+0000 7fc7d37fe640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc7c8034810 0x7fc7c8036cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.005+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7dc04b480 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.005+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fc7c4067180 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.005+0000 7fc7da59c640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc7c8034810 0x7fc7c8036cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.005+0000 7fc7da59c640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc7c8034810 0x7fc7c8036cc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc006fd0 tx=0x7fc7cc006db0 comp rx=0 tx=0).ready entity=mgr.14118 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.009+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc7c403b5a0 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:17.218+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7fc7dc111f20 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.070+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fc7c4066b80 con 0x7fc7dc101b50 2024-06-15T00:48:18.656 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.072+0000 7fc7d37fe640 1 -- 172.21.15.105:0/1864471207 <== mon.0 v2:172.21.15.105:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v9) v1 ==== 88+0+0 (secure 0 0 0) 0x7fc7c404c3a0 con 0x7fc7dc101b50 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc7c8034810 msgr2=0x7fc7c8036cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc7c8034810 0x7fc7c8036cc0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc7cc006fd0 tx=0x7fc7cc006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 msgr2=0x7fc7dc110830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc110830 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7fc7c4031780 tx=0x7fc7c402c040 comp rx=0 tx=0).stop 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 shutdown_connections 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fc7c8034810 0x7fc7c8036cc0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 --2- 172.21.15.105:0/1864471207 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7dc101b50 0x7fc7dc110830 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 >> 172.21.15.105:0/1864471207 conn(0x7fc7dc0fd350 msgr2=0x7fc7dc0fdea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 shutdown_connections 2024-06-15T00:48:18.657 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:18.075+0000 7fc7e123f640 1 -- 172.21.15.105:0/1864471207 wait complete. 2024-06-15T00:48:19.275 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:19 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1864471207' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-06-15T00:48:19.275 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:19 smithi105 ceph-mon[28877]: mgrmap e9: smithi105.oorfym(active, since 32s) 2024-06-15T00:48:19.828 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:48:19.834 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "epoch": 9, 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "available": true, 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "active_name": "smithi105.oorfym", 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "num_standby": 0 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.065+0000 7fb986eb4640 1 Processor -- start 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb986eb4640 1 -- start start 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb986eb4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb9801046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb986eb4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980104c00 con 0x7fb980075c40 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb984c50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb9801046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb984c50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb9801046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43478/0 (socket says 172.21.15.105:43478) 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb984c50640 1 -- 172.21.15.105:0/961118042 learned_addr learned my addr 172.21.15.105:0/961118042 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.066+0000 7fb984c50640 1 -- 172.21.15.105:0/961118042 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb980104d40 con 0x7fb980075c40 2024-06-15T00:48:19.835 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.067+0000 7fb984c50640 1 --2- 172.21.15.105:0/961118042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb9801046c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb974009840 tx=0x7fb9740091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9d97fc65d41994ae server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.067+0000 7fb97f7fe640 1 -- 172.21.15.105:0/961118042 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb974032440 con 0x7fb980075c40 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.067+0000 7fb97f7fe640 1 -- 172.21.15.105:0/961118042 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb974032a50 con 0x7fb980075c40 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.067+0000 7fb97f7fe640 1 -- 172.21.15.105:0/961118042 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb974031aa0 con 0x7fb980075c40 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 -- 172.21.15.105:0/961118042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 msgr2=0x7fb9801046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 --2- 172.21.15.105:0/961118042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb9801046c0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fb974009840 tx=0x7fb9740091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 -- 172.21.15.105:0/961118042 shutdown_connections 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 --2- 172.21.15.105:0/961118042 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb9801046c0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 -- 172.21.15.105:0/961118042 >> 172.21.15.105:0/961118042 conn(0x7fb9800fc150 msgr2=0x7fb9800fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 -- 172.21.15.105:0/961118042 shutdown_connections 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.068+0000 7fb986eb4640 1 -- 172.21.15.105:0/961118042 wait complete. 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb986eb4640 1 Processor -- start 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb986eb4640 1 -- start start 2024-06-15T00:48:19.836 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb986eb4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb980110890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb986eb4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb980110dd0 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb984c50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb980110890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb984c50640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb980110890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43486/0 (socket says 172.21.15.105:43486) 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb984c50640 1 -- 172.21.15.105:0/2583194373 learned_addr learned my addr 172.21.15.105:0/2583194373 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.069+0000 7fb984c50640 1 -- 172.21.15.105:0/2583194373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb974009450 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.070+0000 7fb984c50640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb980110890 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb974031780 tx=0x7fb97402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.072+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb97403b440 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.072+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb97403ba50 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.072+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb980110fd0 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.072+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb97403aa70 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.072+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb980111440 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.073+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fb974039070 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.073+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb98004b480 con 0x7fb980075c40 2024-06-15T00:48:19.837 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.074+0000 7fb97d7fa640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 0x7fb970036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.074+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb97403b7c0 con 0x7fb980075c40 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.074+0000 7fb97ffff640 1 -- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 msgr2=0x7fb970036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2469050433 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.074+0000 7fb97ffff640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 0x7fb970036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.078+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb9740308c0 con 0x7fb980075c40 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.275+0000 7fb97ffff640 1 -- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 msgr2=0x7fb970036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2469050433 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.275+0000 7fb97ffff640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 0x7fb970036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.300+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fb980105ad0 con 0x7fb980075c40 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.301+0000 7fb97d7fa640 1 -- 172.21.15.105:0/2583194373 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v9) v1 ==== 56+0+103 (secure 0 0 0) 0x7fb974037030 con 0x7fb980075c40 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 msgr2=0x7fb970036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 0x7fb970036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 msgr2=0x7fb980110890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:19.838 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb980110890 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb974031780 tx=0x7fb97402c040 comp rx=0 tx=0).stop 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 shutdown_connections 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb9700348b0 0x7fb970036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 --2- 172.21.15.105:0/2583194373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb980075c40 0x7fb980110890 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 >> 172.21.15.105:0/2583194373 conn(0x7fb9800fc150 msgr2=0x7fb9800fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 shutdown_connections 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:19.303+0000 7fb986eb4640 1 -- 172.21.15.105:0/2583194373 wait complete. 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for the mgr to restart... 2024-06-15T00:48:19.839 INFO:teuthology.orchestra.run.smithi105.stdout:Waiting for mgr epoch 9... 2024-06-15T00:48:20.086 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:20 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2583194373' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: Active manager daemon smithi105.oorfym restarted 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: Activating manager daemon smithi105.oorfym 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: osdmap e3: 0 total, 0 up, 0 in 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: mgrmap e10: smithi105.oorfym(active, starting, since 0.00323936s) 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi105.oorfym", "id": "smithi105.oorfym"}]: dispatch 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:48:22.767 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: Manager daemon smithi105.oorfym is now available 2024-06-15T00:48:22.768 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:48:22.768 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:48:22.768 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:48:22.768 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:22 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/trash_purge_schedule"}]: dispatch 2024-06-15T00:48:24.470 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout { 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "mgrmap_epoch": 11, 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout "initialized": true 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout } 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.312+0000 7fb625b61640 1 Processor -- start 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.312+0000 7fb625b61640 1 -- start start 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.313+0000 7fb625b61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.313+0000 7fb625b61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6201012e0 con 0x7fb620100990 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.313+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:24.471 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.313+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43502/0 (socket says 172.21.15.105:43502) 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.313+0000 7fb61f7fe640 1 -- 172.21.15.105:0/2723144236 learned_addr learned my addr 172.21.15.105:0/2723144236 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.313+0000 7fb61f7fe640 1 -- 172.21.15.105:0/2723144236 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb620101420 con 0x7fb620100990 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.314+0000 7fb61f7fe640 1 --2- 172.21.15.105:0/2723144236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620100da0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb610009e50 tx=0x7fb610009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=3c17baaa0361b414 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.314+0000 7fb61e7fc640 1 -- 172.21.15.105:0/2723144236 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610031560 con 0x7fb620100990 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.314+0000 7fb61e7fc640 1 -- 172.21.15.105:0/2723144236 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb61002d050 con 0x7fb620100990 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.314+0000 7fb61e7fc640 1 -- 172.21.15.105:0/2723144236 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610007a20 con 0x7fb620100990 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 -- 172.21.15.105:0/2723144236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 msgr2=0x7fb620100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 --2- 172.21.15.105:0/2723144236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620100da0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fb610009e50 tx=0x7fb610009840 comp rx=0 tx=0).stop 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 -- 172.21.15.105:0/2723144236 shutdown_connections 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 --2- 172.21.15.105:0/2723144236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620100da0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 -- 172.21.15.105:0/2723144236 >> 172.21.15.105:0/2723144236 conn(0x7fb6200fc150 msgr2=0x7fb6200fe580 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:24.472 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 -- 172.21.15.105:0/2723144236 shutdown_connections 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.315+0000 7fb625b61640 1 -- 172.21.15.105:0/2723144236 wait complete. 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb625b61640 1 Processor -- start 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb625b61640 1 -- start start 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb625b61640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620075b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb625b61640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb620113620 con 0x7fb620100990 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620075b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb61f7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620075b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43510/0 (socket says 172.21.15.105:43510) 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.316+0000 7fb61f7fe640 1 -- 172.21.15.105:0/1105243383 learned_addr learned my addr 172.21.15.105:0/1105243383 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb61f7fe640 1 -- 172.21.15.105:0/1105243383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb610009ae0 con 0x7fb620100990 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb61f7fe640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620075b30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb610007700 tx=0x7fb610009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610031a60 con 0x7fb620100990 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb610031bc0 con 0x7fb620100990 2024-06-15T00:48:24.473 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb620113760 con 0x7fb620100990 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb610045570 con 0x7fb620100990 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.317+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb620113b70 con 0x7fb620100990 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.318+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 9) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fb610038050 con 0x7fb620100990 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.318+0000 7fb617fff640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.318+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 --> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb60c037470 con 0x7fb60c0348b0 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.318+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 936+0+0 (secure 0 0 0) 0x7fb610066f10 con 0x7fb620100990 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.318+0000 7fb61effd640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 msgr2=0x7fb60c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2469050433 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.318+0000 7fb61effd640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.519+0000 7fb61effd640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 msgr2=0x7fb60c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2469050433 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.519+0000 7fb61effd640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.920+0000 7fb61effd640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 msgr2=0x7fb60c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2469050433 2024-06-15T00:48:24.474 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:20.920+0000 7fb61effd640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.800000 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:21.721+0000 7fb61effd640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 msgr2=0x7fb60c036d60 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/2469050433 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:21.721+0000 7fb61effd640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 1.600000 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:22.615+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mgrmap(e 10) v1 ==== 41726+0+0 (secure 0 0 0) 0x7fb610043da0 con 0x7fb620100990 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:22.615+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 msgr2=0x7fb60c036d60 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:22.615+0000 7fb617fff640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.616+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7fb61003b9c0 con 0x7fb620100990 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.617+0000 7fb617fff640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb60c0383d0 0x7fb60c03a7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.617+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fb60c037470 con 0x7fb60c0383d0 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.617+0000 7fb61effd640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb60c0383d0 0x7fb60c03a7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.618+0000 7fb61effd640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb60c0383d0 0x7fb60c03a7b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb608003940 tx=0x7fb608009350 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.618+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6549 (secure 0 0 0) 0x7fb60c037470 con 0x7fb60c0383d0 2024-06-15T00:48:24.475 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.623+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fb62004b480 con 0x7fb60c0383d0 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.623+0000 7fb617fff640 1 -- 172.21.15.105:0/1105243383 <== mgr.14160 v2:172.21.15.105:6800/3568068422 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fb62004b480 con 0x7fb60c0383d0 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb60c0383d0 msgr2=0x7fb60c03a7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb60c0383d0 0x7fb60c03a7b0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fb608003940 tx=0x7fb608009350 comp rx=0 tx=0).stop 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 msgr2=0x7fb620075b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620075b30 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7fb610007700 tx=0x7fb610009840 comp rx=0 tx=0).stop 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 shutdown_connections 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb60c0383d0 0x7fb60c03a7b0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:6800/2469050433,v1:172.21.15.105:6801/2469050433] conn(0x7fb60c0348b0 0x7fb60c036d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 --2- 172.21.15.105:0/1105243383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb620100990 0x7fb620075b30 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 >> 172.21.15.105:0/1105243383 conn(0x7fb6200fc150 msgr2=0x7fb6200fcdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 shutdown_connections 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:23.624+0000 7fb625b61640 1 -- 172.21.15.105:0/1105243383 wait complete. 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:mgr epoch 9 is available 2024-06-15T00:48:24.476 INFO:teuthology.orchestra.run.smithi105.stdout:Generating a dashboard self-signed certificate... 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: [15/Jun/2024:00:48:23] ENGINE Bus STARTING 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: mgrmap e11: smithi105.oorfym(active, since 1.0054s) 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: from='client.14164 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: [15/Jun/2024:00:48:23] ENGINE Serving on https://172.21.15.105:7150 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: [15/Jun/2024:00:48:23] ENGINE Bus STARTED 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: [15/Jun/2024:00:48:23] ENGINE Error in HTTPServer.serve 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Traceback (most recent call last): 2024-06-15T00:48:24.712 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._run(expiration_interval) 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: s = self.context.wrap_socket( 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: return self.sslsocket_class._create( 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self.do_handshake() 2024-06-15T00:48:24.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:48:24.714 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._sslobj.do_handshake() 2024-06-15T00:48:24.714 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:48:24.714 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:24.714 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:24 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout Self-signed certificate created 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd26b657640 1 Processor -- start 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd26b657640 1 -- start start 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd26b657640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd2641046c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd26b657640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd264104c00 con 0x7fd264075c40 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd2693f3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd2641046c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:25.709 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd2693f3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd2641046c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42348/0 (socket says 172.21.15.105:42348) 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.884+0000 7fd2693f3640 1 -- 172.21.15.105:0/2538008890 learned_addr learned my addr 172.21.15.105:0/2538008890 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.885+0000 7fd2693f3640 1 -- 172.21.15.105:0/2538008890 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264104d40 con 0x7fd264075c40 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.885+0000 7fd2693f3640 1 --2- 172.21.15.105:0/2538008890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd2641046c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd260009840 tx=0x7fd2600091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9e0b8c7d183d4270 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd25bfff640 1 -- 172.21.15.105:0/2538008890 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd260032440 con 0x7fd264075c40 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd25bfff640 1 -- 172.21.15.105:0/2538008890 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd260032a50 con 0x7fd264075c40 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd25bfff640 1 -- 172.21.15.105:0/2538008890 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd260031aa0 con 0x7fd264075c40 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd26b657640 1 -- 172.21.15.105:0/2538008890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 msgr2=0x7fd2641046c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd26b657640 1 --2- 172.21.15.105:0/2538008890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd2641046c0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fd260009840 tx=0x7fd2600091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd26b657640 1 -- 172.21.15.105:0/2538008890 shutdown_connections 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd26b657640 1 --2- 172.21.15.105:0/2538008890 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd2641046c0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd26b657640 1 -- 172.21.15.105:0/2538008890 >> 172.21.15.105:0/2538008890 conn(0x7fd2640fc190 msgr2=0x7fd2640fe5a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.886+0000 7fd26b657640 1 -- 172.21.15.105:0/2538008890 shutdown_connections 2024-06-15T00:48:25.710 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd26b657640 1 -- 172.21.15.105:0/2538008890 wait complete. 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd26b657640 1 Processor -- start 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd26b657640 1 -- start start 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd26b657640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd264110850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd26b657640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd264110d90 con 0x7fd264075c40 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd2693f3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd264110850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd2693f3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd264110850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42362/0 (socket says 172.21.15.105:42362) 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.887+0000 7fd2693f3640 1 -- 172.21.15.105:0/3186509539 learned_addr learned my addr 172.21.15.105:0/3186509539 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd2693f3640 1 -- 172.21.15.105:0/3186509539 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd260009450 con 0x7fd264075c40 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd2693f3640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd264110850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd260009420 tx=0x7fd26002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26003b440 con 0x7fd264075c40 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd26003ba50 con 0x7fd264075c40 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26003aa80 con 0x7fd264075c40 2024-06-15T00:48:25.711 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd264110f90 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.888+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd264111400 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.889+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 11) v1 ==== 41853+0+0 (secure 0 0 0) 0x7fd260039070 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.889+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd26404b480 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.889+0000 7fd259ffb640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fd250034730 0x7fd250036be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.889+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fd260066c60 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.890+0000 7fd268bf2640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fd250034730 0x7fd250036be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.890+0000 7fd268bf2640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fd250034730 0x7fd250036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd254006fd0 tx=0x7fd254006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:24.894+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd26003abe0 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.052+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fd260039320 con 0x7fd264075c40 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.075+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fd264111400 con 0x7fd250034730 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.152+0000 7fd259ffb640 1 -- 172.21.15.105:0/3186509539 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fd264111400 con 0x7fd250034730 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fd250034730 msgr2=0x7fd250036be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:25.712 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fd250034730 0x7fd250036be0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7fd254006fd0 tx=0x7fd254006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 msgr2=0x7fd264110850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd264110850 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7fd260009420 tx=0x7fd26002c040 comp rx=0 tx=0).stop 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 shutdown_connections 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fd250034730 0x7fd250036be0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 --2- 172.21.15.105:0/3186509539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd264075c40 0x7fd264110850 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 >> 172.21.15.105:0/3186509539 conn(0x7fd2640fc190 msgr2=0x7fd2640fccd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 shutdown_connections 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:25.155+0000 7fd26b657640 1 -- 172.21.15.105:0/3186509539 wait complete. 2024-06-15T00:48:25.713 INFO:teuthology.orchestra.run.smithi105.stdout:Creating initial admin user... 2024-06-15T00:48:26.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:26 smithi105 ceph-mon[28877]: mgrmap e12: smithi105.oorfym(active, since 2s) 2024-06-15T00:48:26.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:26 smithi105 ceph-mon[28877]: from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:26.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:26 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:26.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:26 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:26.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:26 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout {"username": "admin", "password": "$2b$12$z0Gxn1gTF7sBgjFFeBPmY.eVOxkslNV5hEfbtqa5uvU8xW8QNHZNO", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1718412506, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.099+0000 7f68bcb98640 1 Processor -- start 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.099+0000 7f68bcb98640 1 -- start start 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.100+0000 7f68bcb98640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b8104720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.100+0000 7f68bcb98640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8104c60 con 0x7f68b8075be0 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.100+0000 7f68b659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b8104720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:27.606 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.100+0000 7f68b659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b8104720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42366/0 (socket says 172.21.15.105:42366) 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.100+0000 7f68b659c640 1 -- 172.21.15.105:0/3733238357 learned_addr learned my addr 172.21.15.105:0/3733238357 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.100+0000 7f68b659c640 1 -- 172.21.15.105:0/3733238357 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b8104da0 con 0x7f68b8075be0 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68b659c640 1 --2- 172.21.15.105:0/3733238357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b8104720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009e50 tx=0x7f68a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=673fe5bc0b094e7f server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68b559a640 1 -- 172.21.15.105:0/3733238357 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a8031560 con 0x7f68b8075be0 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68b559a640 1 -- 172.21.15.105:0/3733238357 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68a802d050 con 0x7f68b8075be0 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68b559a640 1 -- 172.21.15.105:0/3733238357 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a8007a20 con 0x7f68b8075be0 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68bcb98640 1 -- 172.21.15.105:0/3733238357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 msgr2=0x7f68b8104720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:27.607 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68bcb98640 1 --2- 172.21.15.105:0/3733238357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b8104720 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009e50 tx=0x7f68a8009840 comp rx=0 tx=0).stop 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68bcb98640 1 -- 172.21.15.105:0/3733238357 shutdown_connections 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68bcb98640 1 --2- 172.21.15.105:0/3733238357 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b8104720 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.101+0000 7f68bcb98640 1 -- 172.21.15.105:0/3733238357 >> 172.21.15.105:0/3733238357 conn(0x7f68b80fc150 msgr2=0x7f68b80fe560 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68bcb98640 1 -- 172.21.15.105:0/3733238357 shutdown_connections 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68bcb98640 1 -- 172.21.15.105:0/3733238357 wait complete. 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68bcb98640 1 Processor -- start 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68bcb98640 1 -- start start 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68bcb98640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b81108d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68bcb98640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b8110e10 con 0x7f68b8075be0 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68b659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b81108d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68b659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b81108d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42374/0 (socket says 172.21.15.105:42374) 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.102+0000 7f68b659c640 1 -- 172.21.15.105:0/655037309 learned_addr learned my addr 172.21.15.105:0/655037309 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:27.608 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68b659c640 1 -- 172.21.15.105:0/655037309 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68a8009ae0 con 0x7f68b8075be0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68b659c640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b81108d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009e20 tx=0x7f68a8009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a8031de0 con 0x7f68b8075be0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68a803b8c0 con 0x7f68b8075be0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b8111010 con 0x7f68b8075be0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68a8045770 con 0x7f68b8075be0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.103+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b8111480 con 0x7f68b8075be0 2024-06-15T00:48:27.609 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.104+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f68a8038050 con 0x7f68b8075be0 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.104+0000 7f68aeffd640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f68a40348b0 0x7f68a4036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.104+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b804b480 con 0x7f68b8075be0 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.104+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f68a8067340 con 0x7f68b8075be0 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.104+0000 7f68b5d9b640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f68a40348b0 0x7f68a4036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.104+0000 7f68b5d9b640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f68a40348b0 0x7f68a4036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68a0006fd0 tx=0x7f68a0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.109+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f68a8038e60 con 0x7f68b8075be0 2024-06-15T00:48:27.610 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.287+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f68b805e8c0 con 0x7f68a40348b0 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.513+0000 7f68aeffd640 1 -- 172.21.15.105:0/655037309 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f68b805e8c0 con 0x7f68a40348b0 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f68a40348b0 msgr2=0x7f68a4036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f68a40348b0 0x7f68a4036d60 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f68a0006fd0 tx=0x7f68a0006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 msgr2=0x7f68b81108d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b81108d0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7f68a8009e20 tx=0x7f68a8009840 comp rx=0 tx=0).stop 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 shutdown_connections 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f68a40348b0 0x7f68a4036d60 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 --2- 172.21.15.105:0/655037309 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f68b8075be0 0x7f68b81108d0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 >> 172.21.15.105:0/655037309 conn(0x7f68b80fc150 msgr2=0x7f68b80fcd20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 shutdown_connections 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:26.515+0000 7f68bcb98640 1 -- 172.21.15.105:0/655037309 wait complete. 2024-06-15T00:48:27.611 INFO:teuthology.orchestra.run.smithi105.stdout:Fetching dashboard port number... 2024-06-15T00:48:27.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:27 smithi105 ceph-mon[28877]: from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:27.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:27 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stdout 8443 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3c5e4640 1 Processor -- start 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3c5e4640 1 -- start start 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3c5e4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34101390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3c5e4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f341018d0 con 0x7f7f34100f80 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3a380640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3a380640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42386/0 (socket says 172.21.15.105:42386) 2024-06-15T00:48:28.736 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.001+0000 7f7f3a380640 1 -- 172.21.15.105:0/1457468404 learned_addr learned my addr 172.21.15.105:0/1457468404 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.002+0000 7f7f3a380640 1 -- 172.21.15.105:0/1457468404 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f34101a10 con 0x7f7f34100f80 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.002+0000 7f7f3a380640 1 --2- 172.21.15.105:0/1457468404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34101390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009840 tx=0x7f7f300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=ffbf9d9c7b0e7d35 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3937e640 1 -- 172.21.15.105:0/1457468404 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f30032440 con 0x7f7f34100f80 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3937e640 1 -- 172.21.15.105:0/1457468404 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f30032a50 con 0x7f7f34100f80 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3937e640 1 -- 172.21.15.105:0/1457468404 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f30031aa0 con 0x7f7f34100f80 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/1457468404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 msgr2=0x7f7f34101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 --2- 172.21.15.105:0/1457468404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34101390 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f7f30009840 tx=0x7f7f300091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/1457468404 shutdown_connections 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 --2- 172.21.15.105:0/1457468404 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34101390 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/1457468404 >> 172.21.15.105:0/1457468404 conn(0x7f7f340fb990 msgr2=0x7f7f340fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/1457468404 shutdown_connections 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.003+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/1457468404 wait complete. 2024-06-15T00:48:28.737 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3c5e4640 1 Processor -- start 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3c5e4640 1 -- start start 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3c5e4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34100780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3c5e4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f34113540 con 0x7f7f34100f80 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3a380640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34100780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3a380640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34100780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42398/0 (socket says 172.21.15.105:42398) 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3a380640 1 -- 172.21.15.105:0/3611728870 learned_addr learned my addr 172.21.15.105:0/3611728870 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3a380640 1 -- 172.21.15.105:0/3611728870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f30009450 con 0x7f7f34100f80 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.004+0000 7f7f3a380640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34100780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7f30031780 tx=0x7f7f3002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.005+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f3003b440 con 0x7f7f34100f80 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.005+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f34113680 con 0x7f7f34100f80 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.005+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f3003ba50 con 0x7f7f34100f80 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.005+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f3003aab0 con 0x7f7f34100f80 2024-06-15T00:48:28.738 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.005+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f34113a30 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.005+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7f7f30039070 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.006+0000 7f7f2affd640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7f20034860 0x7f7f20036d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.006+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7f30067250 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.006+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f3405eb20 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.006+0000 7f7f39b7f640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7f20034860 0x7f7f20036d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.006+0000 7f7f39b7f640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7f20034860 0x7f7f20036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f24006fd0 tx=0x7f7f24006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.010+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7f3003f080 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.177+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7f7f3404b4d0 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.177+0000 7f7f2affd640 1 -- 172.21.15.105:0/3611728870 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v8) v1 ==== 112+0+5 (secure 0 0 0) 0x7f7f30037020 con 0x7f7f34100f80 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7f20034860 msgr2=0x7f7f20036d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7f20034860 0x7f7f20036d10 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7f7f24006fd0 tx=0x7f7f24006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:28.739 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 msgr2=0x7f7f34100780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34100780 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f7f30031780 tx=0x7f7f3002c040 comp rx=0 tx=0).stop 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 shutdown_connections 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7f20034860 0x7f7f20036d10 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 --2- 172.21.15.105:0/3611728870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7f34100f80 0x7f7f34100780 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 >> 172.21.15.105:0/3611728870 conn(0x7f7f340fb990 msgr2=0x7f7f340fc400 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 shutdown_connections 2024-06-15T00:48:28.740 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:28.179+0000 7f7f3c5e4640 1 -- 172.21.15.105:0/3611728870 wait complete. 2024-06-15T00:48:28.744 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 1 from systemctl is-enabled firewalld.service 2024-06-15T00:48:28.744 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout disabled 2024-06-15T00:48:28.752 INFO:teuthology.orchestra.run.smithi105.stdout:Non-zero exit code 3 from systemctl is-active firewalld.service 2024-06-15T00:48:28.752 INFO:teuthology.orchestra.run.smithi105.stdout:systemctl: stdout inactive 2024-06-15T00:48:28.752 INFO:teuthology.orchestra.run.smithi105.stdout:firewalld.service is not enabled 2024-06-15T00:48:28.752 INFO:teuthology.orchestra.run.smithi105.stdout:Not possible to open ports <[8443]>. firewalld.service is not available 2024-06-15T00:48:28.753 INFO:teuthology.orchestra.run.smithi105.stdout:Ceph Dashboard is now available at: 2024-06-15T00:48:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: URL: https://smithi105.front.sepia.ceph.com:8443/ 2024-06-15T00:48:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: User: admin 2024-06-15T00:48:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: Password: iti2mqz0gh 2024-06-15T00:48:28.753 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:28.754 INFO:teuthology.orchestra.run.smithi105.stdout:Saving cluster configuration to /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config directory 2024-06-15T00:48:28.754 INFO:teuthology.orchestra.run.smithi105.stdout:Enabling autotune for osd_memory_target 2024-06-15T00:48:28.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:28 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3611728870' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-06-15T00:48:30.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.168+0000 7fb92f647640 1 Processor -- start 2024-06-15T00:48:30.059 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.168+0000 7fb92f647640 1 -- start start 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.168+0000 7fb92f647640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb928101390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.168+0000 7fb92f647640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb9281018d0 con 0x7fb928100f80 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.168+0000 7fb92d3e3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb928101390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.169+0000 7fb92d3e3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb928101390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42410/0 (socket says 172.21.15.105:42410) 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.169+0000 7fb92d3e3640 1 -- 172.21.15.105:0/2966779706 learned_addr learned my addr 172.21.15.105:0/2966779706 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.169+0000 7fb92d3e3640 1 -- 172.21.15.105:0/2966779706 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb928101a10 con 0x7fb928100f80 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.169+0000 7fb92d3e3640 1 --2- 172.21.15.105:0/2966779706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb928101390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb924009840 tx=0x7fb9240091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=cf0f7046c0edecc2 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb91ffff640 1 -- 172.21.15.105:0/2966779706 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb924032440 con 0x7fb928100f80 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb91ffff640 1 -- 172.21.15.105:0/2966779706 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb924032a50 con 0x7fb928100f80 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb91ffff640 1 -- 172.21.15.105:0/2966779706 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb924031aa0 con 0x7fb928100f80 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 -- 172.21.15.105:0/2966779706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 msgr2=0x7fb928101390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:30.060 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 --2- 172.21.15.105:0/2966779706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb928101390 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fb924009840 tx=0x7fb9240091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 -- 172.21.15.105:0/2966779706 shutdown_connections 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 --2- 172.21.15.105:0/2966779706 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb928101390 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 -- 172.21.15.105:0/2966779706 >> 172.21.15.105:0/2966779706 conn(0x7fb9280fb990 msgr2=0x7fb9280fdda0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 -- 172.21.15.105:0/2966779706 shutdown_connections 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.170+0000 7fb92f647640 1 -- 172.21.15.105:0/2966779706 wait complete. 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92f647640 1 Processor -- start 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92f647640 1 -- start start 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92f647640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb92810e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92f647640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb92810eb30 con 0x7fb928100f80 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92d3e3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb92810e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92d3e3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb92810e5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42416/0 (socket says 172.21.15.105:42416) 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92d3e3640 1 -- 172.21.15.105:0/2362627196 learned_addr learned my addr 172.21.15.105:0/2362627196 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:30.061 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92d3e3640 1 -- 172.21.15.105:0/2362627196 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb924009450 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.171+0000 7fb92d3e3640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb92810e5f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb924009420 tx=0x7fb92402c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.172+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb92403b440 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.172+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb92403ba50 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.172+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb92810ed30 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.172+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb92403aa20 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.172+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb92810f1a0 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.173+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 12) v1 ==== 41959+0+0 (secure 0 0 0) 0x7fb924039070 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.173+0000 7fb91dffb640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb914036b00 0x7fb914038fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.173+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb924066fd0 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.173+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb92805ead0 con 0x7fb928100f80 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.173+0000 7fb92cbe2640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb914036b00 0x7fb914038fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:30.062 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.173+0000 7fb92cbe2640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb914036b00 0x7fb914038fb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb918006fd0 tx=0x7fb918006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.177+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb924049020 con 0x7fb928100f80 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.343+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=osd_memory_target_autotune}] v 0) v1 -- 0x7fb928102880 con 0x7fb928100f80 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.343+0000 7fb91dffb640 1 -- 172.21.15.105:0/2362627196 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=osd_memory_target_autotune}]=0 v8)=0 v8) v1 ==== 127+0+0 (secure 0 0 0) 0x7fb9240530c0 con 0x7fb928100f80 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb914036b00 msgr2=0x7fb914038fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb914036b00 0x7fb914038fb0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fb918006fd0 tx=0x7fb918006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 msgr2=0x7fb92810e5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb92810e5f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fb924009420 tx=0x7fb92402c040 comp rx=0 tx=0).stop 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 shutdown_connections 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb914036b00 0x7fb914038fb0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 --2- 172.21.15.105:0/2362627196 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb928100f80 0x7fb92810e5f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 >> 172.21.15.105:0/2362627196 conn(0x7fb9280fb990 msgr2=0x7fb928105440 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 shutdown_connections 2024-06-15T00:48:30.063 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:29.345+0000 7fb92f647640 1 -- 172.21.15.105:0/2362627196 wait complete. 2024-06-15T00:48:30.703 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:30 smithi105 ceph-mon[28877]: mgrmap e13: smithi105.oorfym(active, since 6s) 2024-06-15T00:48:31.274 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.443+0000 7f79bb329640 1 Processor -- start 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79bb329640 1 -- start start 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79bb329640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4076280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79bb329640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4078110 con 0x7f79b4077d00 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79b90c5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4076280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79b90c5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4076280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42432/0 (socket says 172.21.15.105:42432) 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79b90c5640 1 -- 172.21.15.105:0/1537737836 learned_addr learned my addr 172.21.15.105:0/1537737836 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.444+0000 7f79b90c5640 1 -- 172.21.15.105:0/1537737836 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b40767c0 con 0x7f79b4077d00 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.445+0000 7f79b90c5640 1 --2- 172.21.15.105:0/1537737836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4076280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009840 tx=0x7f79b00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=9ca1e83c20a68333 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.445+0000 7f79abfff640 1 -- 172.21.15.105:0/1537737836 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b0032440 con 0x7f79b4077d00 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.445+0000 7f79abfff640 1 -- 172.21.15.105:0/1537737836 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79b0032a50 con 0x7f79b4077d00 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.445+0000 7f79abfff640 1 -- 172.21.15.105:0/1537737836 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b0031aa0 con 0x7f79b4077d00 2024-06-15T00:48:31.275 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- 172.21.15.105:0/1537737836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 msgr2=0x7f79b4076280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 --2- 172.21.15.105:0/1537737836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4076280 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f79b0009840 tx=0x7f79b00091d0 comp rx=0 tx=0).stop 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- 172.21.15.105:0/1537737836 shutdown_connections 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 --2- 172.21.15.105:0/1537737836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4076280 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- 172.21.15.105:0/1537737836 >> 172.21.15.105:0/1537737836 conn(0x7f79b40fd280 msgr2=0x7f79b40ff690 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- 172.21.15.105:0/1537737836 shutdown_connections 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- 172.21.15.105:0/1537737836 wait complete. 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 Processor -- start 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- start start 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4112a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.446+0000 7f79bb329640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79b4112fa0 con 0x7f79b4077d00 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79b90c5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4112a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79b90c5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4112a60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42442/0 (socket says 172.21.15.105:42442) 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79b90c5640 1 -- 172.21.15.105:0/2720148489 learned_addr learned my addr 172.21.15.105:0/2720148489 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:31.276 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79b90c5640 1 -- 172.21.15.105:0/2720148489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79b0009450 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79b90c5640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4112a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f79b0031780 tx=0x7f79b002c040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b003b440 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79b003ba50 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79b41131a0 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79b003aa70 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.447+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79b4113610 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.448+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f79b0039070 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.448+0000 7f79a9ffb640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f79a0034950 0x7f79a0036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.448+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79b404b480 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.448+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f79b0066f90 con 0x7f79b4077d00 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.448+0000 7f79b88c4640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f79a0034950 0x7f79a0036e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.449+0000 7f79b88c4640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f79a0034950 0x7f79a0036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f79a4006fd0 tx=0x7f79a4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:31.277 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.453+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f79b003f080 con 0x7f79b4077d00 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.703+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config-key set, key=mgr/dashboard/cluster/status}] v 0) v1 -- 0x7f79b4107c80 con 0x7f79b4077d00 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.705+0000 7f79a9ffb640 1 -- 172.21.15.105:0/2720148489 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config-key set, key=mgr/dashboard/cluster/status}]=0 set mgr/dashboard/cluster/status v33)=0 set mgr/dashboard/cluster/status v33) v1 ==== 153+0+0 (secure 0 0 0) 0x7f79b0037070 con 0x7f79b4077d00 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr set mgr/dashboard/cluster/status 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f79a0034950 msgr2=0x7f79a0036e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f79a0034950 0x7f79a0036e00 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 crypto rx=0x7f79a4006fd0 tx=0x7f79a4006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 msgr2=0x7f79b4112a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4112a60 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f79b0031780 tx=0x7f79b002c040 comp rx=0 tx=0).stop 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 shutdown_connections 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f79a0034950 0x7f79a0036e00 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 --2- 172.21.15.105:0/2720148489 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f79b4077d00 0x7f79b4112a60 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 >> 172.21.15.105:0/2720148489 conn(0x7f79b40fd280 msgr2=0x7f79b40fddf0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 shutdown_connections 2024-06-15T00:48:31.278 INFO:teuthology.orchestra.run.smithi105.stdout:/usr/bin/ceph: stderr 2024-06-15T00:48:30.707+0000 7f79bb329640 1 -- 172.21.15.105:0/2720148489 wait complete. 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout:You can access the Ceph CLI as following in case of multi-cluster or non-default config: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: sudo /home/ubuntu/cephtest/cephadm shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout:Or, if you are only running a single cluster on this host: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: sudo /home/ubuntu/cephtest/cephadm shell 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout:Please consider enabling telemetry to help improve Ceph: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: ceph telemetry on 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.279 INFO:teuthology.orchestra.run.smithi105.stdout:For more information see: 2024-06-15T00:48:31.280 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.280 INFO:teuthology.orchestra.run.smithi105.stdout: https://docs.ceph.com/docs/master/mgr/telemetry/ 2024-06-15T00:48:31.280 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:31.280 INFO:teuthology.orchestra.run.smithi105.stdout:Bootstrap complete. 2024-06-15T00:48:31.307 INFO:tasks.cephadm:Fetching config... 2024-06-15T00:48:31.307 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:48:31.308 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-06-15T00:48:31.324 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-06-15T00:48:31.324 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:48:31.324 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-06-15T00:48:31.379 INFO:tasks.cephadm:Fetching mon keyring... 2024-06-15T00:48:31.379 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:48:31.379 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/keyring of=/dev/stdout 2024-06-15T00:48:31.443 INFO:tasks.cephadm:Fetching pub ssh key... 2024-06-15T00:48:31.443 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:48:31.443 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-06-15T00:48:31.498 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-06-15T00:48:31.499 DEBUG:teuthology.orchestra.run.smithi105:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCIshCpzeVToblkQEzSASMOxEMm1Rl+Zg2b/0FdhEI2Y63k2FeDjBRMyIDSHXGzUGFeWGUdHrgigi3N1N3C8Fe+9KpcmkstcJ/VqAexW/D0nR0O4Oj0DuWoxNlgQdOdJpsSlIDAX3+G/3B3m9lP3m7f78uwhVD3ZZtwkblq/YkrqumFU2z0EzQ0LER23Ex0DvmMq4JL89humV1jm6Oxc0xRZiFy8DrbSh2ak01IgaHPDAuxzQOxaUM2cf5o4VL9fJKSZt0dzZMxABh9mV/ifhmasRueazSDCmR2gFyHoUA6warz/fumszcvkaMiaNfJ9HzctPUZF/4+Dw++ErHbvQYtE5x2jyFZbVl9+OVAXmDe6r5FP2c0OdobS9Dkd73DzK3kqd9dgST3qfQkK51XGHyRL+9TSIstQ/+T1VM5IUKpoEMj+kRRrRiqBLKfl5jkqTvA54bEXV1g4Y1Aqc9X+BbBZMHjsnNZdhSKF+XSTdGwDboVJa9OSKw/mBn+Ve01s18= ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-15T00:48:31.573 INFO:teuthology.orchestra.run.smithi105.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCIshCpzeVToblkQEzSASMOxEMm1Rl+Zg2b/0FdhEI2Y63k2FeDjBRMyIDSHXGzUGFeWGUdHrgigi3N1N3C8Fe+9KpcmkstcJ/VqAexW/D0nR0O4Oj0DuWoxNlgQdOdJpsSlIDAX3+G/3B3m9lP3m7f78uwhVD3ZZtwkblq/YkrqumFU2z0EzQ0LER23Ex0DvmMq4JL89humV1jm6Oxc0xRZiFy8DrbSh2ak01IgaHPDAuxzQOxaUM2cf5o4VL9fJKSZt0dzZMxABh9mV/ifhmasRueazSDCmR2gFyHoUA6warz/fumszcvkaMiaNfJ9HzctPUZF/4+Dw++ErHbvQYtE5x2jyFZbVl9+OVAXmDe6r5FP2c0OdobS9Dkd73DzK3kqd9dgST3qfQkK51XGHyRL+9TSIstQ/+T1VM5IUKpoEMj+kRRrRiqBLKfl5jkqTvA54bEXV1g4Y1Aqc9X+BbBZMHjsnNZdhSKF+XSTdGwDboVJa9OSKw/mBn+Ve01s18= ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:48:31.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCIshCpzeVToblkQEzSASMOxEMm1Rl+Zg2b/0FdhEI2Y63k2FeDjBRMyIDSHXGzUGFeWGUdHrgigi3N1N3C8Fe+9KpcmkstcJ/VqAexW/D0nR0O4Oj0DuWoxNlgQdOdJpsSlIDAX3+G/3B3m9lP3m7f78uwhVD3ZZtwkblq/YkrqumFU2z0EzQ0LER23Ex0DvmMq4JL89humV1jm6Oxc0xRZiFy8DrbSh2ak01IgaHPDAuxzQOxaUM2cf5o4VL9fJKSZt0dzZMxABh9mV/ifhmasRueazSDCmR2gFyHoUA6warz/fumszcvkaMiaNfJ9HzctPUZF/4+Dw++ErHbvQYtE5x2jyFZbVl9+OVAXmDe6r5FP2c0OdobS9Dkd73DzK3kqd9dgST3qfQkK51XGHyRL+9TSIstQ/+T1VM5IUKpoEMj+kRRrRiqBLKfl5jkqTvA54bEXV1g4Y1Aqc9X+BbBZMHjsnNZdhSKF+XSTdGwDboVJa9OSKw/mBn+Ve01s18= ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-06-15T00:48:31.633 INFO:teuthology.orchestra.run.smithi143.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCIshCpzeVToblkQEzSASMOxEMm1Rl+Zg2b/0FdhEI2Y63k2FeDjBRMyIDSHXGzUGFeWGUdHrgigi3N1N3C8Fe+9KpcmkstcJ/VqAexW/D0nR0O4Oj0DuWoxNlgQdOdJpsSlIDAX3+G/3B3m9lP3m7f78uwhVD3ZZtwkblq/YkrqumFU2z0EzQ0LER23Ex0DvmMq4JL89humV1jm6Oxc0xRZiFy8DrbSh2ak01IgaHPDAuxzQOxaUM2cf5o4VL9fJKSZt0dzZMxABh9mV/ifhmasRueazSDCmR2gFyHoUA6warz/fumszcvkaMiaNfJ9HzctPUZF/4+Dw++ErHbvQYtE5x2jyFZbVl9+OVAXmDe6r5FP2c0OdobS9Dkd73DzK3kqd9dgST3qfQkK51XGHyRL+9TSIstQ/+T1VM5IUKpoEMj+kRRrRiqBLKfl5jkqTvA54bEXV1g4Y1Aqc9X+BbBZMHjsnNZdhSKF+XSTdGwDboVJa9OSKw/mBn+Ve01s18= ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:48:31.645 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-06-15T00:48:31.907 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:48:32.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:31 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2720148489' entity='client.admin' 2024-06-15T00:48:34.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 -- 172.21.15.105:0/859422346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 msgr2=0x7fb00c0fa9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:34.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 --2- 172.21.15.105:0/859422346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c0fa9c0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7faffc004850 tx=0x7faffc009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 -- 172.21.15.105:0/859422346 shutdown_connections 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 --2- 172.21.15.105:0/859422346 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c0fa9c0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 -- 172.21.15.105:0/859422346 >> 172.21.15.105:0/859422346 conn(0x7fb00c0f5d70 msgr2=0x7fb00c0f81a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 -- 172.21.15.105:0/859422346 shutdown_connections 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.168+0000 7fb012493640 1 -- 172.21.15.105:0/859422346 wait complete. 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.169+0000 7fb012493640 1 Processor -- start 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.169+0000 7fb012493640 1 -- start start 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.169+0000 7fb012493640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c10b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.169+0000 7fb012493640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb00c10b8f0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb00bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c10b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:34.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb00bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c10b3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58800/0 (socket says 172.21.15.105:58800) 2024-06-15T00:48:34.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb00bfff640 1 -- 172.21.15.105:0/2924237237 learned_addr learned my addr 172.21.15.105:0/2924237237 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:34.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb00bfff640 1 -- 172.21.15.105:0/2924237237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faffc005040 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb00bfff640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c10b3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faffc0098f0 tx=0x7faffc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:34.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faffc007a20 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb00c10baf0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7faffc033ae0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faffc03dbd0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.170+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb00c10bf60 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.171+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7faffc030020 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.171+0000 7fb008ff9640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7faff4034900 0x7faff4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.171+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb00c05ead0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.171+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7faffc066bf0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.171+0000 7fb00b7fe640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7faff4034900 0x7faff4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:34.173 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.172+0000 7fb00b7fe640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7faff4034900 0x7faff4036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb000006fd0 tx=0x7fb000006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:34.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.176+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7faffc03b370 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.343+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fb00c10bfd0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.346+0000 7fb008ff9640 1 -- 172.21.15.105:0/2924237237 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v9)=0 v9) v1 ==== 125+0+0 (secure 0 0 0) 0x7faffc03f0c0 con 0x7fb00c0fa5b0 2024-06-15T00:48:34.348 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7faff4034900 msgr2=0x7faff4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7faff4034900 0x7faff4036db0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 crypto rx=0x7fb000006fd0 tx=0x7fb000006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 msgr2=0x7fb00c10b3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c10b3b0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7faffc0098f0 tx=0x7faffc009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 shutdown_connections 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7faff4034900 0x7faff4036db0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 --2- 172.21.15.105:0/2924237237 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb00c0fa5b0 0x7fb00c10b3b0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.348+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 >> 172.21.15.105:0/2924237237 conn(0x7fb00c0f5d70 msgr2=0x7fb00c0f6650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.349+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 shutdown_connections 2024-06-15T00:48:34.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:34.349+0000 7fb012493640 1 -- 172.21.15.105:0/2924237237 wait complete. 2024-06-15T00:48:34.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: Deploying daemon crash.smithi105 on smithi105 2024-06-15T00:48:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:34 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2924237237' entity='client.admin' 2024-06-15T00:48:34.974 INFO:tasks.cephadm:Distributing conf and client.admin keyring to all hosts + 0755 2024-06-15T00:48:34.974 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch client-keyring set client.admin '*' --mode 0755 2024-06-15T00:48:35.561 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:48:37.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.545+0000 7fc29c4a7640 1 -- 172.21.15.105:0/37567553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 msgr2=0x7fc2940fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:37.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.545+0000 7fc29c4a7640 1 --2- 172.21.15.105:0/37567553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc2940fb2e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fc290004850 tx=0x7fc290009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:37.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.545+0000 7fc29c4a7640 1 -- 172.21.15.105:0/37567553 shutdown_connections 2024-06-15T00:48:37.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.545+0000 7fc29c4a7640 1 --2- 172.21.15.105:0/37567553 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc2940fb2e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:37.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.545+0000 7fc29c4a7640 1 -- 172.21.15.105:0/37567553 >> 172.21.15.105:0/37567553 conn(0x7fc2940f65f0 msgr2=0x7fc2940f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:37.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.546+0000 7fc29c4a7640 1 -- 172.21.15.105:0/37567553 shutdown_connections 2024-06-15T00:48:37.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.546+0000 7fc29c4a7640 1 -- 172.21.15.105:0/37567553 wait complete. 2024-06-15T00:48:37.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.546+0000 7fc29c4a7640 1 Processor -- start 2024-06-15T00:48:37.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.546+0000 7fc29c4a7640 1 -- start start 2024-06-15T00:48:37.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.546+0000 7fc29c4a7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc29410c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:37.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.546+0000 7fc29c4a7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc29410c960 con 0x7fc2940faed0 2024-06-15T00:48:37.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29a243640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc29410c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:37.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29a243640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc29410c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58814/0 (socket says 172.21.15.105:58814) 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29a243640 1 -- 172.21.15.105:0/3028315905 learned_addr learned my addr 172.21.15.105:0/3028315905 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29a243640 1 -- 172.21.15.105:0/3028315905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc290005040 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29a243640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc29410c420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2900098f0 tx=0x7fc290009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc290007a20 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc290033ae0 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc29410cb60 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc29003dc70 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.547+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc29410cfd0 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.548+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fc290030020 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.548+0000 7fc28affd640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc280034900 0x7fc280036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.548+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc290067d60 con 0x7fc2940faed0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.548+0000 7fc299a42640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc280034900 0x7fc280036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:37.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.548+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc294101260 con 0x7fc2940faed0 2024-06-15T00:48:37.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.549+0000 7fc299a42640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc280034900 0x7fc280036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc284006fd0 tx=0x7fc284006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:37.552 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.553+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc2900463e0 con 0x7fc2940faed0 2024-06-15T00:48:37.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.722+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- mgr_command(tid 0: {"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}) v1 -- 0x7fc294065950 con 0x7fc280034900 2024-06-15T00:48:37.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.725+0000 7fc28affd640 1 -- 172.21.15.105:0/3028315905 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fc294065950 con 0x7fc280034900 2024-06-15T00:48:37.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc280034900 msgr2=0x7fc280036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:37.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc280034900 0x7fc280036db0 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 crypto rx=0x7fc284006fd0 tx=0x7fc284006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:37.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 msgr2=0x7fc29410c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:37.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc29410c420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7fc2900098f0 tx=0x7fc290009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:37.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 shutdown_connections 2024-06-15T00:48:37.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc280034900 0x7fc280036db0 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:37.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 --2- 172.21.15.105:0/3028315905 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc2940faed0 0x7fc29410c420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:37.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.727+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 >> 172.21.15.105:0/3028315905 conn(0x7fc2940f65f0 msgr2=0x7fc294065240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:37.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.728+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 shutdown_connections 2024-06-15T00:48:37.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:37.728+0000 7fc29c4a7640 1 -- 172.21.15.105:0/3028315905 wait complete. 2024-06-15T00:48:38.717 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi143 2024-06-15T00:48:38.717 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:48:38.718 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.conf 2024-06-15T00:48:38.736 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:48:38.736 DEBUG:teuthology.orchestra.run.smithi143:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:48:38.795 INFO:tasks.cephadm:Adding host smithi143 to orchestrator... 2024-06-15T00:48:38.795 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch host add smithi143 2024-06-15T00:48:38.945 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:38 smithi105 ceph-mon[28877]: from='client.14184 -' entity='client.admin' cmd=[{"prefix": "orch client-keyring set", "entity": "client.admin", "placement": "*", "mode": "0755", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:38.945 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:38 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:39.282 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:48:40.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: Deploying daemon node-exporter.smithi105 on smithi105 2024-06-15T00:48:40.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='client.14186 -' entity='client.crash.smithi105' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:40.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='client.14188 -' entity='client.crash.smithi105' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:40.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='client.14190 -' entity='client.crash.smithi105' cmd=[{"prefix": "crash post", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:40.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:40 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:40.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 -- 172.21.15.105:0/1022566164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 msgr2=0x7f1aa00fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:40.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 --2- 172.21.15.105:0/1022566164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa00fb450 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7f1a94004850 tx=0x7f1a94009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:40.966 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 -- 172.21.15.105:0/1022566164 shutdown_connections 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 --2- 172.21.15.105:0/1022566164 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa00fb450 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 -- 172.21.15.105:0/1022566164 >> 172.21.15.105:0/1022566164 conn(0x7f1aa00f6820 msgr2=0x7f1aa00f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 -- 172.21.15.105:0/1022566164 shutdown_connections 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 -- 172.21.15.105:0/1022566164 wait complete. 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 Processor -- start 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.955+0000 7f1aa7493640 1 -- start start 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa7493640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa010c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa7493640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa010cbe0 con 0x7f1aa00fb040 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa522f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa010c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa522f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa010c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58880/0 (socket says 172.21.15.105:58880) 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa522f640 1 -- 172.21.15.105:0/2655330625 learned_addr learned my addr 172.21.15.105:0/2655330625 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa522f640 1 -- 172.21.15.105:0/2655330625 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a94005040 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa522f640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa010c6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009fa0 tx=0x7f1a94009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a94033630 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a94033c40 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa010cde0 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a9403cc60 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.956+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa010d250 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.957+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f1a94030020 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.958+0000 7f1a91ffb640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f1a8c034900 0x7f1a8c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.958+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f1a940662b0 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.958+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1aa010d250 con 0x7f1aa00fb040 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.958+0000 7f1aa4a2e640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f1a8c034900 0x7f1a8c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.958+0000 7f1aa4a2e640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f1a8c034900 0x7f1a8c036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c006fd0 tx=0x7f1a9c006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:40.968 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:40.962+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1aa010d250 con 0x7f1aa00fb040 2024-06-15T00:48:41.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:41.182+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}) v1 -- 0x7f1aa010d250 con 0x7f1a8c034900 2024-06-15T00:48:42.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:42 smithi105 ceph-mon[28877]: from='client.14192 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:43.765 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:43 smithi105 ceph-mon[28877]: Deploying cephadm binary to smithi143 2024-06-15T00:48:43.766 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:43 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:44.070 INFO:teuthology.orchestra.run.smithi105.stdout:Added host 'smithi143' with addr '172.21.15.143' 2024-06-15T00:48:44.071 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.070+0000 7f1a91ffb640 1 -- 172.21.15.105:0/2655330625 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+49 (secure 0 0 0) 0x7f1aa010d250 con 0x7f1a8c034900 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f1a8c034900 msgr2=0x7f1a8c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f1a8c034900 0x7f1a8c036db0 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 crypto rx=0x7f1a9c006fd0 tx=0x7f1a9c006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 msgr2=0x7f1aa010c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa010c6a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f1a94009fa0 tx=0x7f1a94009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 shutdown_connections 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f1a8c034900 0x7f1a8c036db0 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 --2- 172.21.15.105:0/2655330625 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1aa00fb040 0x7f1aa010c6a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 >> 172.21.15.105:0/2655330625 conn(0x7f1aa00f6820 msgr2=0x7f1aa0064f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 shutdown_connections 2024-06-15T00:48:44.073 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:44.073+0000 7f1aa7493640 1 -- 172.21.15.105:0/2655330625 wait complete. 2024-06-15T00:48:44.610 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch host ls --format=json 2024-06-15T00:48:44.891 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:48:45.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:45.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: Deploying daemon alertmanager.smithi105 on smithi105 2024-06-15T00:48:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: Added host smithi143 2024-06-15T00:48:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:45 smithi105 ceph-mon[28877]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:46.237 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:46 smithi105 ceph-mon[28877]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-15T00:48:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 -- 172.21.15.105:0/3832778952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 msgr2=0x7f90ec0fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 --2- 172.21.15.105:0/3832778952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec0fb450 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f90dc004850 tx=0x7f90dc009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 -- 172.21.15.105:0/3832778952 shutdown_connections 2024-06-15T00:48:46.884 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 --2- 172.21.15.105:0/3832778952 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec0fb450 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 -- 172.21.15.105:0/3832778952 >> 172.21.15.105:0/3832778952 conn(0x7f90ec0f6820 msgr2=0x7f90ec0f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 -- 172.21.15.105:0/3832778952 shutdown_connections 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.875+0000 7f90f0fe4640 1 -- 172.21.15.105:0/3832778952 wait complete. 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90f0fe4640 1 Processor -- start 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90f0fe4640 1 -- start start 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90f0fe4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec065230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90f0fe4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90ec10f390 con 0x7f90ec0fb040 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec065230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90ea59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec065230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57914/0 (socket says 172.21.15.105:57914) 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90ea59c640 1 -- 172.21.15.105:0/4044697870 learned_addr learned my addr 172.21.15.105:0/4044697870 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:46.885 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90ea59c640 1 -- 172.21.15.105:0/4044697870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90dc005040 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90ea59c640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec065230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009fa0 tx=0x7f90dc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc033630 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90ec10f4d0 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f90dc033c40 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f90dc03cc60 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.876+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90ec10f8e0 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.877+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f90dc030020 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.877+0000 7f90e2ffd640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f90d8034900 0x7f90d8036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.877+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90ec05ead0 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.877+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f90dc066430 con 0x7f90ec0fb040 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.878+0000 7f90e3fff640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f90d8034900 0x7f90d8036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.878+0000 7f90e3fff640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f90d8034900 0x7f90d8036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90d4006fd0 tx=0x7f90d4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:46.886 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:46.882+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f90dc03d470 con 0x7f90ec0fb040 2024-06-15T00:48:47.079 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.079+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f90ec100db0 con 0x7f90d8034900 2024-06-15T00:48:47.080 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.080+0000 7f90e2ffd640 1 -- 172.21.15.105:0/4044697870 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+161 (secure 0 0 0) 0x7f90ec100db0 con 0x7f90d8034900 2024-06-15T00:48:47.080 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:48:47.080 INFO:teuthology.orchestra.run.smithi105.stdout:[{"addr": "172.21.15.105", "hostname": "smithi105", "labels": [], "status": ""}, {"addr": "172.21.15.143", "hostname": "smithi143", "labels": [], "status": ""}] 2024-06-15T00:48:47.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f90d8034900 msgr2=0x7f90d8036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:47.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f90d8034900 0x7f90d8036db0 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 crypto rx=0x7f90d4006fd0 tx=0x7f90d4006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:47.082 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 msgr2=0x7f90ec065230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec065230 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f90dc009fa0 tx=0x7f90dc009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 shutdown_connections 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f90d8034900 0x7f90d8036db0 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 --2- 172.21.15.105:0/4044697870 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f90ec0fb040 0x7f90ec065230 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 >> 172.21.15.105:0/4044697870 conn(0x7f90ec0f6820 msgr2=0x7f90ec0f72f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 shutdown_connections 2024-06-15T00:48:47.083 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:47.082+0000 7f90f0fe4640 1 -- 172.21.15.105:0/4044697870 wait complete. 2024-06-15T00:48:47.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:47 smithi105 ceph-mon[28877]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:48.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:48 smithi105 ceph-mon[28877]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:48:48.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:48 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:48.481 INFO:tasks.cephadm:Setting crush tunables to default 2024-06-15T00:48:48.482 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd crush tunables default 2024-06-15T00:48:49.007 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:48:49.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:49 smithi105 ceph-mon[28877]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:50.329 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:50 smithi105 ceph-mon[28877]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:52.212 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:52 smithi105 ceph-mon[28877]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:52.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 -- 172.21.15.105:0/2175472865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 msgr2=0x7f63d40fd990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:52.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 --2- 172.21.15.105:0/2175472865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d40fd990 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f63d0004850 tx=0x7f63d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:52.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 -- 172.21.15.105:0/2175472865 shutdown_connections 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 --2- 172.21.15.105:0/2175472865 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d40fd990 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 -- 172.21.15.105:0/2175472865 >> 172.21.15.105:0/2175472865 conn(0x7f63d40f5da0 msgr2=0x7f63d40f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 -- 172.21.15.105:0/2175472865 shutdown_connections 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.714+0000 7f63db2c4640 1 -- 172.21.15.105:0/2175472865 wait complete. 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63db2c4640 1 Processor -- start 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63db2c4640 1 -- start start 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63db2c4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d410c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63db2c4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d410c9e0 con 0x7f63d40fd580 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63d9060640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d410c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63d9060640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d410c4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:57928/0 (socket says 172.21.15.105:57928) 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.715+0000 7f63d9060640 1 -- 172.21.15.105:0/1616808768 learned_addr learned my addr 172.21.15.105:0/1616808768 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:48:52.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63d9060640 1 -- 172.21.15.105:0/1616808768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63d0005040 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63d9060640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d410c4a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009fa0 tx=0x7f63d0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63d0033630 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63d0033c40 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d410cbe0 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63d003cc60 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.716+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d410d050 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.717+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f63d0030020 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.717+0000 7f63c9ffb640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f63c0034900 0x7f63c0036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.717+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f63d0066330 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.717+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d4102370 con 0x7f63d40fd580 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.717+0000 7f63d885f640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f63c0034900 0x7f63c0036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.718+0000 7f63d885f640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f63c0034900 0x7f63c0036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f63c4006fd0 tx=0x7f63c4006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:52.732 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.722+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f63d00323d0 con 0x7f63d40fd580 2024-06-15T00:48:52.897 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:52.897+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f63d405ead0 con 0x7f63d40fd580 2024-06-15T00:48:53.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.186+0000 7f63c9ffb640 1 -- 172.21.15.105:0/1616808768 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v4) v1 ==== 124+0+0 (secure 0 0 0) 0x7f63d405ead0 con 0x7f63d40fd580 2024-06-15T00:48:53.186 INFO:teuthology.orchestra.run.smithi105.stderr:adjusted tunables profile to default 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.188+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f63c0034900 msgr2=0x7f63c0036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.188+0000 7f63db2c4640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f63c0034900 0x7f63c0036db0 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 crypto rx=0x7f63c4006fd0 tx=0x7f63c4006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 msgr2=0x7f63d410c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d410c4a0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f63d0009fa0 tx=0x7f63d0009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 shutdown_connections 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f63c0034900 0x7f63c0036db0 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 --2- 172.21.15.105:0/1616808768 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f63d40fd580 0x7f63d410c4a0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 >> 172.21.15.105:0/1616808768 conn(0x7f63d40f5da0 msgr2=0x7f63d4102f30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 shutdown_connections 2024-06-15T00:48:53.189 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:48:53.189+0000 7f63db2c4640 1 -- 172.21.15.105:0/1616808768 wait complete. 2024-06-15T00:48:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:53 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1616808768' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-15T00:48:53.911 INFO:tasks.cephadm:Adding mon.smithi105 on smithi105 2024-06-15T00:48:53.911 INFO:tasks.cephadm:Adding mon.smithi143 on smithi143 2024-06-15T00:48:53.911 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch apply mon '2;smithi105:172.21.15.105=smithi105;smithi143:172.21.15.143=smithi143' 2024-06-15T00:48:54.159 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:48:54.207 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:48:54.440 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:54 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1616808768' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-15T00:48:54.440 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:54 smithi105 ceph-mon[28877]: osdmap e4: 0 total, 0 up, 0 in 2024-06-15T00:48:54.441 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:54 smithi105 ceph-mon[28877]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:55.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 -- 172.21.15.143:0/3511351837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 msgr2=0x7fc4580feb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:55.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 --2- 172.21.15.143:0/3511351837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc4580feb40 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7fc448004850 tx=0x7fc448009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:55.737 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 -- 172.21.15.143:0/3511351837 shutdown_connections 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 --2- 172.21.15.143:0/3511351837 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc4580feb40 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 -- 172.21.15.143:0/3511351837 >> 172.21.15.143:0/3511351837 conn(0x7fc4580f65f0 msgr2=0x7fc4580f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 -- 172.21.15.143:0/3511351837 shutdown_connections 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.734+0000 7fc45c8d9640 1 -- 172.21.15.143:0/3511351837 wait complete. 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45c8d9640 1 Processor -- start 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45c8d9640 1 -- start start 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45c8d9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc45810c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45c8d9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45810c9b0 con 0x7fc458065f00 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc45810c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45659c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc45810c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:41856/0 (socket says 172.21.15.143:41856) 2024-06-15T00:48:55.738 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45659c640 1 -- 172.21.15.143:0/1771348140 learned_addr learned my addr 172.21.15.143:0/1771348140 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.735+0000 7fc45659c640 1 -- 172.21.15.143:0/1771348140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc448005040 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.736+0000 7fc45659c640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc45810c470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc448003080 tx=0x7fc448009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.736+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc448033630 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.736+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc45810cbb0 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.736+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc448033c40 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.736+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc44803dc50 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.736+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45810d020 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.737+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fc448030020 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.737+0000 7fc44effd640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc444034900 0x7fc444036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.737+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc4480667a0 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.737+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc458102360 con 0x7fc458065f00 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.737+0000 7fc44ffff640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc444034900 0x7fc444036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:55.739 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.738+0000 7fc44ffff640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc444034900 0x7fc444036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc440006fd0 tx=0x7fc440006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:55.741 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.742+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc448044d20 con 0x7fc458065f00 2024-06-15T00:48:55.913 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.913+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 --> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi105:172.21.15.105=smithi105;smithi143:172.21.15.143=smithi143", "target": ["mon-mgr", ""]}) v1 -- 0x7fc45810d740 con 0x7fc444034900 2024-06-15T00:48:55.918 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.918+0000 7fc44effd640 1 -- 172.21.15.143:0/1771348140 <== mgr.14160 v2:172.21.15.105:6800/3568068422 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc45810d740 con 0x7fc444034900 2024-06-15T00:48:55.918 INFO:teuthology.orchestra.run.smithi143.stdout:Scheduled mon update... 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc444034900 msgr2=0x7fc444036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc444034900 0x7fc444036db0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 crypto rx=0x7fc440006fd0 tx=0x7fc440006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 msgr2=0x7fc45810c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc45810c470 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7fc448003080 tx=0x7fc448009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 shutdown_connections 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc444034900 0x7fc444036db0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 --2- 172.21.15.143:0/1771348140 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc458065f00 0x7fc45810c470 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:55.920 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 >> 172.21.15.143:0/1771348140 conn(0x7fc4580f65f0 msgr2=0x7fc4580f6ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:55.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 shutdown_connections 2024-06-15T00:48:55.921 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:55.920+0000 7fc45c8d9640 1 -- 172.21.15.143:0/1771348140 wait complete. 2024-06-15T00:48:56.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:56.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:55 smithi105 ceph-mon[28877]: Deploying daemon grafana.smithi105 on smithi105 2024-06-15T00:48:56.499 DEBUG:teuthology.orchestra.run.smithi143:mon.smithi143> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi143.service 2024-06-15T00:48:56.501 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:48:56.502 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:48:56.778 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:48:56.826 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:48:57.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:56 smithi105 ceph-mon[28877]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi105:172.21.15.105=smithi105;smithi143:172.21.15.143=smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:48:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:56 smithi105 ceph-mon[28877]: Saving service mon spec with placement smithi105:172.21.15.105=smithi105;smithi143:172.21.15.143=smithi143;count:2 2024-06-15T00:48:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:56 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:56 smithi105 ceph-mon[28877]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:58.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 -- 172.21.15.143:0/537292245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 msgr2=0x7f46900fc830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:58.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 --2- 172.21.15.143:0/537292245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fc830 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7f4680004850 tx=0x7f4680009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:58.285 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 -- 172.21.15.143:0/537292245 shutdown_connections 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 --2- 172.21.15.143:0/537292245 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fc830 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 -- 172.21.15.143:0/537292245 >> 172.21.15.143:0/537292245 conn(0x7f46900f5da0 msgr2=0x7f46900f81b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 -- 172.21.15.143:0/537292245 shutdown_connections 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 -- 172.21.15.143:0/537292245 wait complete. 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 Processor -- start 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.284+0000 7f4694ed2640 1 -- start start 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f4694ed2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fabf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f4694ed2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f469010f180 con 0x7f46900fc420 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f468e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fabf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f468e59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fabf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:42300/0 (socket says 172.21.15.143:42300) 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f468e59c640 1 -- 172.21.15.143:0/205449406 learned_addr learned my addr 172.21.15.143:0/205449406 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:48:58.286 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f468e59c640 1 -- 172.21.15.143:0/205449406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4680005040 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f468e59c640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fabf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46800098f0 tx=0x7f4680009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4680007a20 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4680033ae0 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f468003dc70 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.285+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f469010f2c0 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.286+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f469010f730 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.286+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f4680030020 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.286+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f469005ead0 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.287+0000 7f467effd640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f4678034900 0x7f4678036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.287+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f4680067020 con 0x7f46900fc420 2024-06-15T00:48:58.288 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.287+0000 7f468dd9b640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f4678034900 0x7f4678036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:48:58.289 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.287+0000 7f468dd9b640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f4678034900 0x7f4678036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4684006fd0 tx=0x7f4684006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:48:58.291 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.291+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4680045960 con 0x7f46900fc420 2024-06-15T00:48:58.517 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.516+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f469004b480 con 0x7f46900fc420 2024-06-15T00:48:58.518 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.517+0000 7f467effd640 1 -- 172.21.15.143:0/205449406 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f468003de60 con 0x7f46900fc420 2024-06-15T00:48:58.518 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:48:58.518 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:48:58.518 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f4678034900 msgr2=0x7f4678036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f4678034900 0x7f4678036db0 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 crypto rx=0x7f4684006fd0 tx=0x7f4684006db0 comp rx=0 tx=0).stop 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 msgr2=0x7f46900fabf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fabf0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f46800098f0 tx=0x7f4680009da0 comp rx=0 tx=0).stop 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 shutdown_connections 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f4678034900 0x7f4678036db0 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 --2- 172.21.15.143:0/205449406 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f46900fc420 0x7f46900fabf0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 >> 172.21.15.143:0/205449406 conn(0x7f46900f5da0 msgr2=0x7f46900f65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 shutdown_connections 2024-06-15T00:48:58.520 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:48:58.519+0000 7f4694ed2640 1 -- 172.21.15.143:0/205449406 wait complete. 2024-06-15T00:48:58.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:58 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:48:58.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:58 smithi105 ceph-mon[28877]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:48:58.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:48:58 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/205449406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:00.102 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:00.102 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:00.349 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:00.397 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:00.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:00 smithi105 ceph-mon[28877]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.812+0000 7fc38067e640 1 -- 172.21.15.143:0/3345215211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 msgr2=0x7fc3780fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.812+0000 7fc38067e640 1 --2- 172.21.15.143:0/3345215211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc3780fc340 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7fc374004850 tx=0x7fc374009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.812+0000 7fc38067e640 1 -- 172.21.15.143:0/3345215211 shutdown_connections 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.812+0000 7fc38067e640 1 --2- 172.21.15.143:0/3345215211 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc3780fc340 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.812+0000 7fc38067e640 1 -- 172.21.15.143:0/3345215211 >> 172.21.15.143:0/3345215211 conn(0x7fc3780f7710 msgr2=0x7fc3780f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc38067e640 1 -- 172.21.15.143:0/3345215211 shutdown_connections 2024-06-15T00:49:01.815 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc38067e640 1 -- 172.21.15.143:0/3345215211 wait complete. 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc38067e640 1 Processor -- start 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc38067e640 1 -- start start 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc38067e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc37810c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc38067e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc37810c990 con 0x7fc3780fbf30 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.813+0000 7fc37e41a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc37810c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc37e41a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc37810c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:42310/0 (socket says 172.21.15.143:42310) 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc37e41a640 1 -- 172.21.15.143:0/1402730181 learned_addr learned my addr 172.21.15.143:0/1402730181 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc37e41a640 1 -- 172.21.15.143:0/1402730181 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc374005040 con 0x7fc3780fbf30 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc37e41a640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc37810c450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc374009fa0 tx=0x7fc374009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc374033630 con 0x7fc3780fbf30 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc374033c40 con 0x7fc3780fbf30 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc37810cb90 con 0x7fc3780fbf30 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc37403cc60 con 0x7fc3780fbf30 2024-06-15T00:49:01.816 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.814+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc37810d000 con 0x7fc3780fbf30 2024-06-15T00:49:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.815+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fc374030020 con 0x7fc3780fbf30 2024-06-15T00:49:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.815+0000 7fc36effd640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc364034900 0x7fc364036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.815+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc37805ead0 con 0x7fc3780fbf30 2024-06-15T00:49:01.817 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.815+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fc374066350 con 0x7fc3780fbf30 2024-06-15T00:49:01.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.815+0000 7fc37dc19640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc364034900 0x7fc364036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:01.818 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.816+0000 7fc37dc19640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc364034900 0x7fc364036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc368006fd0 tx=0x7fc368006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:01.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:01.820+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc37403c3d0 con 0x7fc3780fbf30 2024-06-15T00:49:02.046 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.046+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc37804b480 con 0x7fc3780fbf30 2024-06-15T00:49:02.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.046+0000 7fc36effd640 1 -- 172.21.15.143:0/1402730181 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fc374042250 con 0x7fc3780fbf30 2024-06-15T00:49:02.047 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:02.047 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:02.047 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:02.048 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc364034900 msgr2=0x7fc364036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc364034900 0x7fc364036db0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fc368006fd0 tx=0x7fc368006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 msgr2=0x7fc37810c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc37810c450 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7fc374009fa0 tx=0x7fc374009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 shutdown_connections 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fc364034900 0x7fc364036db0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 --2- 172.21.15.143:0/1402730181 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc3780fbf30 0x7fc37810c450 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 >> 172.21.15.143:0/1402730181 conn(0x7fc3780f7710 msgr2=0x7fc3780f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 shutdown_connections 2024-06-15T00:49:02.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:02.048+0000 7fc38067e640 1 -- 172.21.15.143:0/1402730181 wait complete. 2024-06-15T00:49:02.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:02 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1402730181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:03.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:03 smithi105 ceph-mon[28877]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:03.616 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:03.616 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:03.858 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:03.905 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:05.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 -- 172.21.15.143:0/407616322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 msgr2=0x7fdbcc0fc340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:05.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 --2- 172.21.15.143:0/407616322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc0fc340 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc004850 tx=0x7fdbbc009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:05.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 -- 172.21.15.143:0/407616322 shutdown_connections 2024-06-15T00:49:05.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 --2- 172.21.15.143:0/407616322 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc0fc340 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:05.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 -- 172.21.15.143:0/407616322 >> 172.21.15.143:0/407616322 conn(0x7fdbcc0f7710 msgr2=0x7fdbcc0f9b20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:05.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 -- 172.21.15.143:0/407616322 shutdown_connections 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.249+0000 7fdbd12e5640 1 -- 172.21.15.143:0/407616322 wait complete. 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbd12e5640 1 Processor -- start 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbd12e5640 1 -- start start 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbd12e5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc10c450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbd12e5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbcc10c990 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc10c450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbcaffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc10c450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:42338/0 (socket says 172.21.15.143:42338) 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbcaffd640 1 -- 172.21.15.143:0/3304414264 learned_addr learned my addr 172.21.15.143:0/3304414264 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.250+0000 7fdbcaffd640 1 -- 172.21.15.143:0/3304414264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbbc005040 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.251+0000 7fdbcaffd640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc10c450 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009fa0 tx=0x7fdbbc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.251+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbbc033630 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.251+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbcc10cb90 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.251+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbbc033c40 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.251+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbbc03cc60 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.251+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbcc10d000 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.252+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fdbbc030020 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.252+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbcc05ead0 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.252+0000 7fdbb37fe640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fdbb4034900 0x7fdbb4036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.252+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fdbbc0662b0 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.252+0000 7fdbca7fc640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fdbb4034900 0x7fdbb4036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.252+0000 7fdbca7fc640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fdbb4034900 0x7fdbb4036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006fd0 tx=0x7fdbc0006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:05.309 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.256+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdbbc0432f0 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:05 smithi105 ceph-mon[28877]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:05.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.483+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbcc04b480 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.484 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.484+0000 7fdbb37fe640 1 -- 172.21.15.143:0/3304414264 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fdbbc0432f0 con 0x7fdbcc0fbf30 2024-06-15T00:49:05.485 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:05.485 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:05.485 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fdbb4034900 msgr2=0x7fdbb4036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fdbb4034900 0x7fdbb4036db0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fdbc0006fd0 tx=0x7fdbc0006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 msgr2=0x7fdbcc10c450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc10c450 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7fdbbc009fa0 tx=0x7fdbbc009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 shutdown_connections 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fdbb4034900 0x7fdbb4036db0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:05.487 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 --2- 172.21.15.143:0/3304414264 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbcc0fbf30 0x7fdbcc10c450 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:05.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 >> 172.21.15.143:0/3304414264 conn(0x7fdbcc0f7710 msgr2=0x7fdbcc0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:05.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 shutdown_connections 2024-06-15T00:49:05.488 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:05.486+0000 7fdbd12e5640 1 -- 172.21.15.143:0/3304414264 wait complete. 2024-06-15T00:49:06.395 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:06 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/3304414264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:07.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:07.046 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:07.294 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:07.341 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:07.509 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:07 smithi105 ceph-mon[28877]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:08.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 -- 172.21.15.143:0/2644369566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 msgr2=0x7fb9300fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:08.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 --2- 172.21.15.143:0/2644369566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb9300fc320 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7fb920004850 tx=0x7fb920009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:08.820 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 -- 172.21.15.143:0/2644369566 shutdown_connections 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 --2- 172.21.15.143:0/2644369566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb9300fc320 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 -- 172.21.15.143:0/2644369566 >> 172.21.15.143:0/2644369566 conn(0x7fb9300f76f0 msgr2=0x7fb9300f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 -- 172.21.15.143:0/2644369566 shutdown_connections 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 -- 172.21.15.143:0/2644369566 wait complete. 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 Processor -- start 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.814+0000 7fb93529d640 1 -- start start 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb93529d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb93010c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:08.821 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb93529d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb93010c9b0 con 0x7fb9300fbf10 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb92effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb93010c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb92effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb93010c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:48208/0 (socket says 172.21.15.143:48208) 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb92effd640 1 -- 172.21.15.143:0/2881269516 learned_addr learned my addr 172.21.15.143:0/2881269516 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb92effd640 1 -- 172.21.15.143:0/2881269516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb920005040 con 0x7fb9300fbf10 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb92effd640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb93010c470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb920009fa0 tx=0x7fb920009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb920033630 con 0x7fb9300fbf10 2024-06-15T00:49:08.822 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93010cbb0 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb920033c40 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.815+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb92003cc60 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.816+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb93010d020 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.816+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7fb920030020 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.816+0000 7fb927fff640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb91c034900 0x7fb91c036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.816+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93005ead0 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.817+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7fb920066350 con 0x7fb9300fbf10 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.817+0000 7fb9267fc640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb91c034900 0x7fb91c036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:08.823 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.817+0000 7fb9267fc640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb91c034900 0x7fb91c036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb918006fd0 tx=0x7fb918006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:08.824 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:08.821+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb92003c3d0 con 0x7fb9300fbf10 2024-06-15T00:49:09.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.046+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb93004b480 con 0x7fb9300fbf10 2024-06-15T00:49:09.047 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.047+0000 7fb927fff640 1 -- 172.21.15.143:0/2881269516 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fb92003bb20 con 0x7fb9300fbf10 2024-06-15T00:49:09.047 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:09.048 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:09.048 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:09.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb91c034900 msgr2=0x7fb91c036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:09.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb91c034900 0x7fb91c036db0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fb918006fd0 tx=0x7fb918006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:09.049 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 msgr2=0x7fb93010c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb93010c470 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7fb920009fa0 tx=0x7fb920009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 shutdown_connections 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7fb91c034900 0x7fb91c036db0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 --2- 172.21.15.143:0/2881269516 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb9300fbf10 0x7fb93010c470 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 >> 172.21.15.143:0/2881269516 conn(0x7fb9300f76f0 msgr2=0x7fb9300f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 shutdown_connections 2024-06-15T00:49:09.050 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:09.049+0000 7fb93529d640 1 -- 172.21.15.143:0/2881269516 wait complete. 2024-06-15T00:49:09.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:09.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:09.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:09.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:09.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:09 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/2881269516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:10.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:10 smithi105 ceph-mon[28877]: Deploying daemon prometheus.smithi105 on smithi105 2024-06-15T00:49:10.617 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:10.618 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:10.862 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:10.909 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:11.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:11 smithi105 ceph-mon[28877]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:12.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 -- 172.21.15.143:0/1580102027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 msgr2=0x7f7b780feb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:12.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 --2- 172.21.15.143:0/1580102027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780feb90 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f7b74004850 tx=0x7f7b74009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:12.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 -- 172.21.15.143:0/1580102027 shutdown_connections 2024-06-15T00:49:12.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 --2- 172.21.15.143:0/1580102027 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780feb90 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:12.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 -- 172.21.15.143:0/1580102027 >> 172.21.15.143:0/1580102027 conn(0x7f7b780f65e0 msgr2=0x7f7b780f8a10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:12.374 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 -- 172.21.15.143:0/1580102027 shutdown_connections 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.369+0000 7f7b8031b640 1 -- 172.21.15.143:0/1580102027 wait complete. 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b8031b640 1 Processor -- start 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b8031b640 1 -- start start 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b8031b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780fc4c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b8031b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b780fca00 con 0x7f7b78065ee0 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b7e0b7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780fc4c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b7e0b7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780fc4c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:48228/0 (socket says 172.21.15.143:48228) 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.370+0000 7f7b7e0b7640 1 -- 172.21.15.143:0/2030540556 learned_addr learned my addr 172.21.15.143:0/2030540556 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b7e0b7640 1 -- 172.21.15.143:0/2030540556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b74005040 con 0x7f7b78065ee0 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b7e0b7640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780fc4c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400af90 tx=0x7f7b74009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b74007a20 con 0x7f7b78065ee0 2024-06-15T00:49:12.375 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7b74032b00 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b780faa90 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7b7403dbb0 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.371+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b780faf00 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.372+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f7b74030020 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.372+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b78102360 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.372+0000 7f7b6effd640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7b64034900 0x7f7b64036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.372+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f7b74066fb0 con 0x7f7b78065ee0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.372+0000 7f7b7d8b6640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7b64034900 0x7f7b64036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:12.376 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.373+0000 7f7b7d8b6640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7b64034900 0x7f7b64036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006fd0 tx=0x7f7b68006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:12.377 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.377+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7b74098df0 con 0x7f7b78065ee0 2024-06-15T00:49:12.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.601+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7b7805ead0 con 0x7f7b78065ee0 2024-06-15T00:49:12.601 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.602+0000 7f7b6effd640 1 -- 172.21.15.143:0/2030540556 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f7b74045ce0 con 0x7f7b78065ee0 2024-06-15T00:49:12.602 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:12.602 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:12.602 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:12.603 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7b64034900 msgr2=0x7f7b64036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7b64034900 0x7f7b64036db0 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 crypto rx=0x7f7b68006fd0 tx=0x7f7b68006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 msgr2=0x7f7b780fc4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780fc4c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7f7b7400af90 tx=0x7f7b74009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 shutdown_connections 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f7b64034900 0x7f7b64036db0 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 --2- 172.21.15.143:0/2030540556 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7b78065ee0 0x7f7b780fc4c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 >> 172.21.15.143:0/2030540556 conn(0x7f7b780f65e0 msgr2=0x7f7b780f6ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 shutdown_connections 2024-06-15T00:49:12.604 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:12.604+0000 7f7b8031b640 1 -- 172.21.15.143:0/2030540556 wait complete. 2024-06-15T00:49:13.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:13 smithi105 ceph-mon[28877]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:13.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:13 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/2030540556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:13.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:13 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:14.194 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:14.195 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:14.442 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:14.491 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:15.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:15 smithi105 ceph-mon[28877]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:15.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 -- 172.21.15.143:0/4095401527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 msgr2=0x7f9efc0fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:15.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 --2- 172.21.15.143:0/4095401527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc0fc320 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009bd0 tx=0x7f9ef4004d10 comp rx=0 tx=0).stop 2024-06-15T00:49:15.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 -- 172.21.15.143:0/4095401527 shutdown_connections 2024-06-15T00:49:15.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 --2- 172.21.15.143:0/4095401527 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc0fc320 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:15.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 -- 172.21.15.143:0/4095401527 >> 172.21.15.143:0/4095401527 conn(0x7f9efc0f76f0 msgr2=0x7f9efc0f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:15.886 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 -- 172.21.15.143:0/4095401527 shutdown_connections 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.885+0000 7f9f02d1e640 1 -- 172.21.15.143:0/4095401527 wait complete. 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f02d1e640 1 Processor -- start 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f02d1e640 1 -- start start 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f02d1e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc10c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f02d1e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9efc10c9b0 con 0x7f9efc0fbf10 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f00aba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc10c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f00aba640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc10c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:48250/0 (socket says 172.21.15.143:48250) 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.886+0000 7f9f00aba640 1 -- 172.21.15.143:0/80807275 learned_addr learned my addr 172.21.15.143:0/80807275 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9f00aba640 1 -- 172.21.15.143:0/80807275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef40098b0 con 0x7f9efc0fbf10 2024-06-15T00:49:15.887 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9f00aba640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc10c470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009ba0 tx=0x7f9ef4004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:15.888 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef40328a0 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ef403d420 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9efc10cbb0 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ef4031590 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.887+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9efc10d740 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.888+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f9ef402f030 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.888+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9efc10d080 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.888+0000 7f9ef97fa640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9ee80348b0 0x7f9ee8036d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.888+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9ef4067b70 con 0x7f9efc0fbf10 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.888+0000 7f9efbfff640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9ee80348b0 0x7f9ee8036d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:15.889 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.889+0000 7f9efbfff640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9ee80348b0 0x7f9ee8036d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9eec006fd0 tx=0x7f9eec006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:15.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:15.893+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9efc10d080 con 0x7f9efc0fbf10 2024-06-15T00:49:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.120+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9efc102340 con 0x7f9efc0fbf10 2024-06-15T00:49:16.120 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.121+0000 7f9ef97fa640 1 -- 172.21.15.143:0/80807275 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9ef4041030 con 0x7f9efc0fbf10 2024-06-15T00:49:16.121 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:16.121 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:16.121 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.123+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9ee80348b0 msgr2=0x7f9ee8036d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.123+0000 7f9f02d1e640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9ee80348b0 0x7f9ee8036d60 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7f9eec006fd0 tx=0x7f9eec006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.123+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 msgr2=0x7f9efc10c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:16.123 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.123+0000 7f9f02d1e640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc10c470 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7f9ef4009ba0 tx=0x7f9ef4004d10 comp rx=0 tx=0).stop 2024-06-15T00:49:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.124+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 shutdown_connections 2024-06-15T00:49:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.124+0000 7f9f02d1e640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9ee80348b0 0x7f9ee8036d60 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.124+0000 7f9f02d1e640 1 --2- 172.21.15.143:0/80807275 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9efc0fbf10 0x7f9efc10c470 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.124+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 >> 172.21.15.143:0/80807275 conn(0x7f9efc0f76f0 msgr2=0x7f9efc0f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.124+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 shutdown_connections 2024-06-15T00:49:16.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:16.124+0000 7f9f02d1e640 1 -- 172.21.15.143:0/80807275 wait complete. 2024-06-15T00:49:17.220 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:17 smithi105 ceph-mon[28877]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:17.220 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:17 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/80807275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:17.693 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:17.693 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:17.941 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:17.988 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:19.152 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:19 smithi105 ceph-mon[28877]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.343+0000 7f9f8b84b640 1 -- 172.21.15.143:0/3255541746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 msgr2=0x7f9f840fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.343+0000 7f9f8b84b640 1 --2- 172.21.15.143:0/3255541746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f840fb2e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f9f80004850 tx=0x7f9f80009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.343+0000 7f9f8b84b640 1 -- 172.21.15.143:0/3255541746 shutdown_connections 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.343+0000 7f9f8b84b640 1 --2- 172.21.15.143:0/3255541746 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f840fb2e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.343+0000 7f9f8b84b640 1 -- 172.21.15.143:0/3255541746 >> 172.21.15.143:0/3255541746 conn(0x7f9f840f65f0 msgr2=0x7f9f840f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f8b84b640 1 -- 172.21.15.143:0/3255541746 shutdown_connections 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f8b84b640 1 -- 172.21.15.143:0/3255541746 wait complete. 2024-06-15T00:49:19.357 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f8b84b640 1 Processor -- start 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f8b84b640 1 -- start start 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f8b84b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f8410c420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f8b84b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8410c960 con 0x7f9f840faed0 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.344+0000 7f9f895e7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f8410c420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f895e7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f8410c420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:47420/0 (socket says 172.21.15.143:47420) 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f895e7640 1 -- 172.21.15.143:0/654770661 learned_addr learned my addr 172.21.15.143:0/654770661 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f895e7640 1 -- 172.21.15.143:0/654770661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f80005040 con 0x7f9f840faed0 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f895e7640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f8410c420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009fa0 tx=0x7f9f80009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f80033630 con 0x7f9f840faed0 2024-06-15T00:49:19.358 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8410cb60 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9f80033c40 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f8003cc60 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.345+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8410cfd0 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.346+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 13) v1 ==== 42003+0+0 (secure 0 0 0) 0x7f9f80030020 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.346+0000 7f9f79ffb640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9f70034900 0x7f9f70036db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.346+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f9f80066440 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.346+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f84101260 con 0x7f9f840faed0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.346+0000 7f9f88de6640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9f70034900 0x7f9f70036db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.347+0000 7f9f88de6640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9f70034900 0x7f9f70036db0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006fd0 tx=0x7f9f74006db0 comp rx=0 tx=0).ready entity=mgr.14160 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:19.359 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.351+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9f800323d0 con 0x7f9f840faed0 2024-06-15T00:49:19.576 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.576+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9f8405ead0 con 0x7f9f840faed0 2024-06-15T00:49:19.577 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.577+0000 7f9f79ffb640 1 -- 172.21.15.143:0/654770661 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f9f8405ead0 con 0x7f9f840faed0 2024-06-15T00:49:19.577 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:19.577 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:19.578 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:19.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9f70034900 msgr2=0x7f9f70036db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:19.579 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9f70034900 0x7f9f70036db0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f9f74006fd0 tx=0x7f9f74006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 msgr2=0x7f9f8410c420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f8410c420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7f9f80009fa0 tx=0x7f9f80009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 shutdown_connections 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f9f70034900 0x7f9f70036db0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 --2- 172.21.15.143:0/654770661 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9f840faed0 0x7f9f8410c420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 >> 172.21.15.143:0/654770661 conn(0x7f9f840f65f0 msgr2=0x7f9f84065240 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 shutdown_connections 2024-06-15T00:49:19.580 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:19.580+0000 7f9f8b84b640 1 -- 172.21.15.143:0/654770661 wait complete. 2024-06-15T00:49:20.412 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:20 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:20.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:20 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:20.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:20 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:20.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:20 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-15T00:49:20.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:20 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/654770661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:20.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:20 smithi105 ceph-mon[28877]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:21.304 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:21.304 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:21.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:21 smithi105 ceph-mon[28877]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-15T00:49:21.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:21 smithi105 ceph-mon[28877]: mgrmap e14: smithi105.oorfym(active, since 57s) 2024-06-15T00:49:21.553 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:21.601 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:22.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 -- 172.21.15.143:0/3444964465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 msgr2=0x7f6b940fc320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:22.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 --2- 172.21.15.143:0/3444964465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b940fc320 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7f6b84004850 tx=0x7f6b84009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:22.935 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 -- 172.21.15.143:0/3444964465 shutdown_connections 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 --2- 172.21.15.143:0/3444964465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b940fc320 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 -- 172.21.15.143:0/3444964465 >> 172.21.15.143:0/3444964465 conn(0x7f6b940f76f0 msgr2=0x7f6b940f9b00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 -- 172.21.15.143:0/3444964465 shutdown_connections 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.921+0000 7f6b9b7df640 1 -- 172.21.15.143:0/3444964465 wait complete. 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9b7df640 1 Processor -- start 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9b7df640 1 -- start start 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9b7df640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b9410c470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9b7df640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b9410c9b0 con 0x7f6b940fbf10 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9957b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b9410c470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9957b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b9410c470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:47432/0 (socket says 172.21.15.143:47432) 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.922+0000 7f6b9957b640 1 -- 172.21.15.143:0/2036438902 learned_addr learned my addr 172.21.15.143:0/2036438902 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:22.936 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b9957b640 1 -- 172.21.15.143:0/2036438902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b84005040 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b9957b640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b9410c470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009fa0 tx=0x7f6b84009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b84033630 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b9410cbb0 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6b84033c40 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6b8403cc60 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.923+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b9410d020 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.924+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b9405ead0 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.924+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 14) v1 ==== 42017+0+0 (secure 0 0 0) 0x7f6b84030020 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.924+0000 7f6b89ffb640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 0x7f6b80036e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.924+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1151+0+0 (secure 0 0 0) 0x7f6b840663d0 con 0x7f6b940fbf10 2024-06-15T00:49:22.937 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.924+0000 7f6b98d7a640 1 -- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 msgr2=0x7f6b80036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3568068422 2024-06-15T00:49:22.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.924+0000 7f6b98d7a640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 0x7f6b80036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:49:22.938 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:22.929+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6b8403bb60 con 0x7f6b940fbf10 2024-06-15T00:49:23.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.125+0000 7f6b98d7a640 1 -- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 msgr2=0x7f6b80036e00 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.105:6800/3568068422 2024-06-15T00:49:23.124 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.125+0000 7f6b98d7a640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 0x7f6b80036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.400000 2024-06-15T00:49:23.153 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.154+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6b9404b480 con 0x7f6b940fbf10 2024-06-15T00:49:23.154 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.154+0000 7f6b89ffb640 1 -- 172.21.15.143:0/2036438902 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f6b8403bb60 con 0x7f6b940fbf10 2024-06-15T00:49:23.154 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:23.154 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:23.154 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 msgr2=0x7f6b80036e00 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 0x7f6b80036e00 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 msgr2=0x7f6b9410c470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b9410c470 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7f6b84009fa0 tx=0x7f6b84009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 shutdown_connections 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:6800/3568068422,v1:172.21.15.105:6801/3568068422] conn(0x7f6b80034950 0x7f6b80036e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 --2- 172.21.15.143:0/2036438902 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f6b940fbf10 0x7f6b9410c470 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 >> 172.21.15.143:0/2036438902 conn(0x7f6b940f76f0 msgr2=0x7f6b940f7f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:23.156 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 shutdown_connections 2024-06-15T00:49:23.157 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:23.156+0000 7f6b9b7df640 1 -- 172.21.15.143:0/2036438902 wait complete. 2024-06-15T00:49:23.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:23 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/2036438902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:24.756 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:24.756 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:24.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: Active manager daemon smithi105.oorfym restarted 2024-06-15T00:49:24.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: Activating manager daemon smithi105.oorfym 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: osdmap e5: 0 total, 0 up, 0 in 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: mgrmap e15: smithi105.oorfym(active, starting, since 0.00560466s) 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi105.oorfym", "id": "smithi105.oorfym"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: Manager daemon smithi105.oorfym is now available 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:49:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:25.011 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:25.061 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:25.861 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:25.861 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:49:25.861 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:49:25.861 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/trash_purge_schedule"}]: dispatch 2024-06-15T00:49:25.861 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:25.862 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:25 smithi105 ceph-mon[28877]: mgrmap e16: smithi105.oorfym(active, since 1.00609s) 2024-06-15T00:49:26.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 -- 172.21.15.143:0/1585735876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 msgr2=0x7fcd9c0fb2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:26.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 --2- 172.21.15.143:0/1585735876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c0fb2e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7fcd90004850 tx=0x7fcd90009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:26.569 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 -- 172.21.15.143:0/1585735876 shutdown_connections 2024-06-15T00:49:26.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 --2- 172.21.15.143:0/1585735876 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c0fb2e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:26.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 -- 172.21.15.143:0/1585735876 >> 172.21.15.143:0/1585735876 conn(0x7fcd9c0f65f0 msgr2=0x7fcd9c0f8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:26.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 -- 172.21.15.143:0/1585735876 shutdown_connections 2024-06-15T00:49:26.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.567+0000 7fcda2a97640 1 -- 172.21.15.143:0/1585735876 wait complete. 2024-06-15T00:49:26.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda2a97640 1 Processor -- start 2024-06-15T00:49:26.570 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda2a97640 1 -- start start 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda2a97640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c065d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda2a97640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd9c10f1f0 con 0x7fcd9c0faed0 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda0833640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c065d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda0833640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c065d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:58312/0 (socket says 172.21.15.143:58312) 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.568+0000 7fcda0833640 1 -- 172.21.15.143:0/1504860674 learned_addr learned my addr 172.21.15.143:0/1504860674 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcda0833640 1 -- 172.21.15.143:0/1504860674 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd90005040 con 0x7fcd9c0faed0 2024-06-15T00:49:26.571 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcda0833640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c065d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fa0 tx=0x7fcd90009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:26.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd90033630 con 0x7fcd9c0faed0 2024-06-15T00:49:26.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd9c10f330 con 0x7fcd9c0faed0 2024-06-15T00:49:26.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd90033c40 con 0x7fcd9c0faed0 2024-06-15T00:49:26.572 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd9003cc60 con 0x7fcd9c0faed0 2024-06-15T00:49:26.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.569+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd9c10f740 con 0x7fcd9c0faed0 2024-06-15T00:49:26.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.570+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 16) v1 ==== 41867+0+0 (secure 0 0 0) 0x7fcd90030020 con 0x7fcd9c0faed0 2024-06-15T00:49:26.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.570+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd9c05ead0 con 0x7fcd9c0faed0 2024-06-15T00:49:26.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.570+0000 7fcd997fa640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd8c034790 0x7fcd8c036c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:26.574 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.571+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fcd900663c0 con 0x7fcd9c0faed0 2024-06-15T00:49:26.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.571+0000 7fcd9bfff640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd8c034790 0x7fcd8c036c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:26.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.571+0000 7fcd9bfff640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd8c034790 0x7fcd8c036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcd88006fd0 tx=0x7fcd88006db0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:26.575 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.575+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcd9003d470 con 0x7fcd9c0faed0 2024-06-15T00:49:26.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.807+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd9c04b480 con 0x7fcd9c0faed0 2024-06-15T00:49:26.807 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.807+0000 7fcd997fa640 1 -- 172.21.15.143:0/1504860674 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fcd9003b370 con 0x7fcd9c0faed0 2024-06-15T00:49:26.807 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:26.808 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:26.808 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:26.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd8c034790 msgr2=0x7fcd8c036c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:26.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd8c034790 0x7fcd8c036c40 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0x7fcd88006fd0 tx=0x7fcd88006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:26.809 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 msgr2=0x7fcd9c065d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c065d80 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7fcd90009fa0 tx=0x7fcd90009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 shutdown_connections 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd8c034790 0x7fcd8c036c40 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 --2- 172.21.15.143:0/1504860674 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd9c0faed0 0x7fcd9c065d80 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 >> 172.21.15.143:0/1504860674 conn(0x7fcd9c0f65f0 msgr2=0x7fcd9c101e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 shutdown_connections 2024-06-15T00:49:26.810 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:26.810+0000 7fcda2a97640 1 -- 172.21.15.143:0/1504860674 wait complete. 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: [15/Jun/2024:00:49:25] ENGINE Bus STARTING 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: [15/Jun/2024:00:49:25] ENGINE Serving on https://172.21.15.105:7150 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: [15/Jun/2024:00:49:25] ENGINE Bus STARTED 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: [15/Jun/2024:00:49:25] ENGINE Error in HTTPServer.serve 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: Traceback (most recent call last): 2024-06-15T00:49:26.997 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._run(expiration_interval) 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: s = self.context.wrap_socket( 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: return self.sslsocket_class._create( 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:49:26.998 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self.do_handshake() 2024-06-15T00:49:26.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:49:26.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: self._sslobj.do_handshake() 2024-06-15T00:49:26.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:49:26.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:26.999 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:28.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:27 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1504860674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:28.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:27 smithi105 ceph-mon[28877]: mgrmap e17: smithi105.oorfym(active, since 2s) 2024-06-15T00:49:28.415 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:28.415 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:28.910 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:28.965 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /etc/ceph/ceph.conf 2024-06-15T00:49:30.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:29 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:29 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:29 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:29 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:29 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:49:30.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 -- 172.21.15.143:0/636946130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 msgr2=0x7f02300fb450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:30.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 --2- 172.21.15.143:0/636946130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f02300fb450 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f0220004850 tx=0x7f0220009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:30.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 -- 172.21.15.143:0/636946130 shutdown_connections 2024-06-15T00:49:30.960 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 --2- 172.21.15.143:0/636946130 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f02300fb450 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 -- 172.21.15.143:0/636946130 >> 172.21.15.143:0/636946130 conn(0x7f02300f6820 msgr2=0x7f02300f8c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 -- 172.21.15.143:0/636946130 shutdown_connections 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 -- 172.21.15.143:0/636946130 wait complete. 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 Processor -- start 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.960+0000 7f02352f1640 1 -- start start 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f02352f1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f023010c6a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f02352f1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f023010cbe0 con 0x7f02300fb040 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f022effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f023010c6a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f022effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f023010c6a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:58326/0 (socket says 172.21.15.143:58326) 2024-06-15T00:49:30.961 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f022effd640 1 -- 172.21.15.143:0/3152228922 learned_addr learned my addr 172.21.15.143:0/3152228922 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f022effd640 1 -- 172.21.15.143:0/3152228922 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0220005040 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.961+0000 7f022effd640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f023010c6a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0220009fa0 tx=0x7f0220009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.962+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0220033630 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.962+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f023010cde0 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.962+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0220033c40 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.962+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f023010d250 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.962+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f022003cc60 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.962+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f0220030020 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.963+0000 7f02277fe640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f021c034a40 0x7f021c036ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.963+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f0220067670 con 0x7f02300fb040 2024-06-15T00:49:30.963 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.963+0000 7f0227fff640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f021c034a40 0x7f021c036ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:30.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.963+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0230101470 con 0x7f02300fb040 2024-06-15T00:49:30.964 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.963+0000 7f0227fff640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f021c034a40 0x7f021c036ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0218006fd0 tx=0x7f0218006db0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:30.967 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:30.967+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f022003a2d0 con 0x7f02300fb040 2024-06-15T00:49:31.211 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.212+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f023005ead0 con 0x7f02300fb040 2024-06-15T00:49:31.212 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.212+0000 7f02277fe640 1 -- 172.21.15.143:0/3152228922 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f022004b090 con 0x7f02300fb040 2024-06-15T00:49:31.212 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:31.212 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:31.212 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f021c034a40 msgr2=0x7f021c036ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f021c034a40 0x7f021c036ef0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0x7f0218006fd0 tx=0x7f0218006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 msgr2=0x7f023010c6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f023010c6a0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f0220009fa0 tx=0x7f0220009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 shutdown_connections 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f021c034a40 0x7f021c036ef0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:31.214 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.214+0000 7f02352f1640 1 --2- 172.21.15.143:0/3152228922 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02300fb040 0x7f023010c6a0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:31.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.215+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 >> 172.21.15.143:0/3152228922 conn(0x7f02300f6820 msgr2=0x7f0230064f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:31.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.215+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 shutdown_connections 2024-06-15T00:49:31.215 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:31.215+0000 7f02352f1640 1 -- 172.21.15.143:0/3152228922 wait complete. 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi105:/etc/ceph/ceph.conf 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi143:/etc/ceph/ceph.conf 2024-06-15T00:49:31.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: Updating smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/3152228922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:49:31.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-15T00:49:31.779 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:31 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:32.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:32 smithi105 ceph-mon[28877]: Deploying daemon crash.smithi143 on smithi143 2024-06-15T00:49:32.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:32.818 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:33.066 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 -- 172.21.15.143:0/1494691034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 msgr2=0x7f5494102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 --2- 172.21.15.143:0/1494691034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f5494102330 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7f5484004850 tx=0x7f5484009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 -- 172.21.15.143:0/1494691034 shutdown_connections 2024-06-15T00:49:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 --2- 172.21.15.143:0/1494691034 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f5494102330 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 -- 172.21.15.143:0/1494691034 >> 172.21.15.143:0/1494691034 conn(0x7f54940f7b30 msgr2=0x7f54940f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:34.657 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 -- 172.21.15.143:0/1494691034 shutdown_connections 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.645+0000 7f549a3b2640 1 -- 172.21.15.143:0/1494691034 wait complete. 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f549a3b2640 1 Processor -- start 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f549a3b2640 1 -- start start 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f549a3b2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f54941124d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f549a3b2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494112a10 con 0x7f5494101f60 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f5493fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f54941124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f5493fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f54941124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:58344/0 (socket says 172.21.15.143:58344) 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f5493fff640 1 -- 172.21.15.143:0/3900660788 learned_addr learned my addr 172.21.15.143:0/3900660788 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.646+0000 7f5493fff640 1 -- 172.21.15.143:0/3900660788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5484005040 con 0x7f5494101f60 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.647+0000 7f5493fff640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f54941124d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5484003080 tx=0x7f5484009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:34.658 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.647+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f548402cf10 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.647+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5484033ae0 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.647+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5494112c10 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.647+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f548403db60 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.647+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5494113080 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.648+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7f5484030020 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.648+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54940fd6d0 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.648+0000 7f54917fa640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f54800349f0 0x7f5480036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.648+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f5484066af0 con 0x7f5494101f60 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.648+0000 7f548bfff640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f54800349f0 0x7f5480036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.649+0000 7f548bfff640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f54800349f0 0x7f5480036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f547c006fd0 tx=0x7f547c006db0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:34.659 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.653+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5484044520 con 0x7f5494101f60 2024-06-15T00:49:34.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.890+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f549404b480 con 0x7f5494101f60 2024-06-15T00:49:34.890 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.890+0000 7f54917fa640 1 -- 172.21.15.143:0/3900660788 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7f5484044340 con 0x7f5494101f60 2024-06-15T00:49:34.891 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:34.891 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:34.891 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f54800349f0 msgr2=0x7f5480036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f54800349f0 0x7f5480036ea0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 crypto rx=0x7f547c006fd0 tx=0x7f547c006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 msgr2=0x7f54941124d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f54941124d0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f5484003080 tx=0x7f5484009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 shutdown_connections 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f54800349f0 0x7f5480036ea0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 --2- 172.21.15.143:0/3900660788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5494101f60 0x7f54941124d0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:34.892 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 >> 172.21.15.143:0/3900660788 conn(0x7f54940f7b30 msgr2=0x7f54940f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:34.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.892+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 shutdown_connections 2024-06-15T00:49:34.893 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:34.893+0000 7f549a3b2640 1 -- 172.21.15.143:0/3900660788 wait complete. 2024-06-15T00:49:35.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:34 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/3900660788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:36.696 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:36.696 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:36.944 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:37.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:37.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:37.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:37.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:36 smithi105 ceph-mon[28877]: Deploying daemon node-exporter.smithi143 on smithi143 2024-06-15T00:49:39.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- 172.21.15.143:0/1389888987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 msgr2=0x7fd77c102330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:39.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 --2- 172.21.15.143:0/1389888987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c102330 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7fd76c004850 tx=0x7fd76c009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:39.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- 172.21.15.143:0/1389888987 shutdown_connections 2024-06-15T00:49:39.179 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 --2- 172.21.15.143:0/1389888987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c102330 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- 172.21.15.143:0/1389888987 >> 172.21.15.143:0/1389888987 conn(0x7fd77c0f7b30 msgr2=0x7fd77c0f9f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- 172.21.15.143:0/1389888987 shutdown_connections 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- 172.21.15.143:0/1389888987 wait complete. 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 Processor -- start 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- start start 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c1124d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.109+0000 7fd781519640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd77c112a10 con 0x7fd77c101f60 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd77affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c1124d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd77affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c1124d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:52046/0 (socket says 172.21.15.143:52046) 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd77affd640 1 -- 172.21.15.143:0/2226973720 learned_addr learned my addr 172.21.15.143:0/2226973720 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd77affd640 1 -- 172.21.15.143:0/2226973720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd76c005040 con 0x7fd77c101f60 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd77affd640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c1124d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd76c003080 tx=0x7fd76c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd76c02cf10 con 0x7fd77c101f60 2024-06-15T00:49:39.180 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd76c033ae0 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd77c112c10 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd76c03db60 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.110+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd77c113080 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.111+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7fd76c030020 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.111+0000 7fd7737fe640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7680349f0 0x7fd768036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.111+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd77c0fd6d0 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.112+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fd76c066b70 con 0x7fd77c101f60 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.112+0000 7fd773fff640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7680349f0 0x7fd768036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.112+0000 7fd773fff640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7680349f0 0x7fd768036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd764006fd0 tx=0x7fd764006db0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:39.181 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.116+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd76c0463f0 con 0x7fd77c101f60 2024-06-15T00:49:39.352 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.352+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd77c04b480 con 0x7fd77c101f60 2024-06-15T00:49:39.353 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.352+0000 7fd7737fe640 1 -- 172.21.15.143:0/2226973720 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fd76c044300 con 0x7fd77c101f60 2024-06-15T00:49:39.353 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:39.353 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:39.354 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7680349f0 msgr2=0x7fd768036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7680349f0 0x7fd768036ea0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0x7fd764006fd0 tx=0x7fd764006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 msgr2=0x7fd77c1124d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c1124d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7fd76c003080 tx=0x7fd76c009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 shutdown_connections 2024-06-15T00:49:39.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7680349f0 0x7fd768036ea0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 --2- 172.21.15.143:0/2226973720 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd77c101f60 0x7fd77c1124d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 >> 172.21.15.143:0/2226973720 conn(0x7fd77c0f7b30 msgr2=0x7fd77c0f9270 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 shutdown_connections 2024-06-15T00:49:39.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:39.354+0000 7fd781519640 1 -- 172.21.15.143:0/2226973720 wait complete. 2024-06-15T00:49:39.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:39 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/2226973720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:40.959 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:40.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:49:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi143.hhtfap", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:49:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi143.hhtfap", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-15T00:49:41.029 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:49:41.029 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:41.029 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:40 smithi105 ceph-mon[28877]: Deploying daemon mgr.smithi143.hhtfap on smithi143 2024-06-15T00:49:41.543 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 -- 172.21.15.143:0/812850077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 msgr2=0x7fef20102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 --2- 172.21.15.143:0/812850077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef20102310 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 crypto rx=0x7fef1c004850 tx=0x7fef1c009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 -- 172.21.15.143:0/812850077 shutdown_connections 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 --2- 172.21.15.143:0/812850077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef20102310 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 -- 172.21.15.143:0/812850077 >> 172.21.15.143:0/812850077 conn(0x7fef20070e10 msgr2=0x7fef20071210 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 -- 172.21.15.143:0/812850077 shutdown_connections 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 -- 172.21.15.143:0/812850077 wait complete. 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 Processor -- start 2024-06-15T00:49:43.354 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.345+0000 7fef27fa4640 1 -- start start 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef27fa4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef201124c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef27fa4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef20112a00 con 0x7fef20101f40 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef25d40640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef201124c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef25d40640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef201124c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.143:52074/0 (socket says 172.21.15.143:52074) 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef25d40640 1 -- 172.21.15.143:0/1228870141 learned_addr learned my addr 172.21.15.143:0/1228870141 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef25d40640 1 -- 172.21.15.143:0/1228870141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef1c005040 con 0x7fef20101f40 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.346+0000 7fef25d40640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef201124c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef1c003080 tx=0x7fef1c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.347+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef1c02cf10 con 0x7fef20101f40 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.347+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef20112c00 con 0x7fef20101f40 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.347+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef1c033ae0 con 0x7fef20101f40 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.347+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef1c03db60 con 0x7fef20101f40 2024-06-15T00:49:43.355 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.347+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef20113070 con 0x7fef20101f40 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.348+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 17) v1 ==== 42061+0+0 (secure 0 0 0) 0x7fef1c030020 con 0x7fef20101f40 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.348+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef200fd6b0 con 0x7fef20101f40 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.348+0000 7fef167fc640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fef0c0349f0 0x7fef0c036ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.348+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fef1c067ac0 con 0x7fef20101f40 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.348+0000 7fef2553f640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fef0c0349f0 0x7fef0c036ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.349+0000 7fef2553f640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fef0c0349f0 0x7fef0c036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10006db0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:43.356 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.353+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fef1c03a6e0 con 0x7fef20101f40 2024-06-15T00:49:43.607 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.606+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef2005eb20 con 0x7fef20101f40 2024-06-15T00:49:43.608 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.607+0000 7fef167fc640 1 -- 172.21.15.143:0/1228870141 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+743 (secure 0 0 0) 0x7fef1c03a2d0 con 0x7fef20101f40 2024-06-15T00:49:43.608 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:43.608 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":1,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:47:22.686810Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-06-15T00:49:43.608 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 1 2024-06-15T00:49:43.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fef0c0349f0 msgr2=0x7fef0c036ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:43.610 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fef0c0349f0 0x7fef0c036ea0 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 crypto rx=0x7fef10006fd0 tx=0x7fef10006db0 comp rx=0 tx=0).stop 2024-06-15T00:49:43.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 msgr2=0x7fef201124c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:43.611 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef201124c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 crypto rx=0x7fef1c003080 tx=0x7fef1c009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 shutdown_connections 2024-06-15T00:49:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fef0c0349f0 0x7fef0c036ea0 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 --2- 172.21.15.143:0/1228870141 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fef20101f40 0x7fef201124c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.609+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 >> 172.21.15.143:0/1228870141 conn(0x7fef20070e10 msgr2=0x7fef200f9cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.610+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 shutdown_connections 2024-06-15T00:49:43.614 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:43.610+0000 7fef27fa4640 1 -- 172.21.15.143:0/1228870141 wait complete. 2024-06-15T00:49:44.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: Deploying daemon mon.smithi143 on smithi143 2024-06-15T00:49:44.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:44 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1228870141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:46.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:45 smithi105 ceph-mon[28877]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:46.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:45 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:46.232 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-06-15T00:49:46.232 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mon dump -f json 2024-06-15T00:49:46.577 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:46 smithi143 systemd[1]: Starting Ceph mon.smithi143 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:49:46.827 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:46 smithi143 podman[35828]: 2024-06-15 00:49:46.498074222 +0000 UTC m=+0.024334208 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:49:46.827 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:46 smithi143 podman[35828]: 2024-06-15 00:49:46.622809082 +0000 UTC m=+0.149069072 container create 54e49738eb43ccb7b97d6e089acfa52a9966c6436c41a6ec8f1876ca9ffa08cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True) 2024-06-15T00:49:47.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 podman[35828]: 2024-06-15 00:49:47.000281647 +0000 UTC m=+0.526541648 container init 54e49738eb43ccb7b97d6e089acfa52a9966c6436c41a6ec8f1876ca9ffa08cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 podman[35828]: 2024-06-15 00:49:47.003708915 +0000 UTC m=+0.529968906 container start 54e49738eb43ccb7b97d6e089acfa52a9966c6436c41a6ec8f1876ca9ffa08cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143, ceph=True, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b) 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: set uid:gid to 167:167 (ceph:ceph) 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: ceph version 17.2.7-1008-ge47ff4b8 (e47ff4b87f747658fc7376cc7065fc56801eb37b) quincy (stable), process ceph-mon, pid 2 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pidfile_write: ignore empty --pid-file 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: load: jerasure load: lrc 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: RocksDB version: 6.15.5 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Compile date Jun 14 2024 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: DB SUMMARY 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: DB Session ID: U6K3OHJBIDL7WC40RE16 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: CURRENT file: CURRENT 2024-06-15T00:49:47.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: IDENTITY file: IDENTITY 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: MANIFEST file: MANIFEST-000003 size: 57 Bytes 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi143/store.db dir, Total Num: 0, files: 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi143/store.db: 000004.log size: 511 ; 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.error_if_exists: 0 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.create_if_missing: 0 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.paranoid_checks: 1 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.track_and_verify_wals_in_manifest: 0 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.env: 0x55638e858440 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.fs: Posix File System 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.info_log: 0x55639101b140 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_file_opening_threads: 16 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.statistics: (nil) 2024-06-15T00:49:47.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.use_fsync: 0 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_log_file_size: 0 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_manifest_file_size: 1073741824 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.log_file_time_to_roll: 0 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.keep_log_file_num: 1000 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.recycle_log_file_num: 0 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.allow_fallocate: 1 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.allow_mmap_reads: 0 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.allow_mmap_writes: 0 2024-06-15T00:49:47.140 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.use_direct_reads: 0 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.create_missing_column_families: 0 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.db_log_dir: 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi143/store.db 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.table_cache_numshardbits: 6 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.WAL_ttl_seconds: 0 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.WAL_size_limit_MB: 0 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.manifest_preallocation_size: 4194304 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.is_fd_close_on_exec: 1 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.advise_random_on_open: 1 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.db_write_buffer_size: 0 2024-06-15T00:49:47.141 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.write_buffer_manager: 0x5563910d87e0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.access_hint_on_compaction_start: 1 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.random_access_max_buffer_size: 1048576 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.use_adaptive_mutex: 0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.rate_limiter: (nil) 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.wal_recovery_mode: 2 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.enable_thread_tracking: 0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.enable_pipelined_write: 0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.unordered_write: 0 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.allow_concurrent_memtable_write: 1 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-06-15T00:49:47.142 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.write_thread_max_yield_usec: 100 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.write_thread_slow_yield_usec: 3 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.row_cache: None 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.wal_filter: None 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.avoid_flush_during_recovery: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.allow_ingest_behind: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.preserve_deletes: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.two_write_queues: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.manual_wal_flush: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.atomic_flush: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.persist_stats_to_disk: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.write_dbid_to_manifest: 0 2024-06-15T00:49:47.143 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.log_readahead_size: 0 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.file_checksum_gen_factory: Unknown 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.best_efforts_recovery: 0 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bgerror_resume_count: 2147483647 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bgerror_resume_retry_interval: 1000000 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.allow_data_in_errors: 0 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.db_host_id: __hostname__ 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_background_jobs: 2 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_background_compactions: -1 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_subcompactions: 1 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.avoid_flush_during_shutdown: 0 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.delayed_write_rate : 16777216 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_total_wal_size: 0 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-06-15T00:49:47.144 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.stats_dump_period_sec: 600 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.stats_persist_period_sec: 600 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.stats_history_buffer_size: 1048576 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_open_files: -1 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bytes_per_sync: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.wal_bytes_per_sync: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.strict_bytes_per_sync: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_readahead_size: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_background_flushes: -1 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Compression algorithms supported: 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kZSTD supported: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kXpressCompression supported: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kBZip2Compression supported: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kZSTDNotFinalCompression supported: 0 2024-06-15T00:49:47.145 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kLZ4Compression supported: 1 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kZlibCompression supported: 1 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kLZ4HCCompression supported: 1 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: kSnappyCompression supported: 1 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Fast CRC32 supported: Supported on x86 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/version_set.cc:4724] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi143/store.db/MANIFEST-000003 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/column_family.cc:595] --------------- Options for column family [default]: 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.merge_operator: 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_filter: None 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_filter_factory: None 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.sst_partitioner_factory: None 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.memtable_factory: SkipListFactory 2024-06-15T00:49:47.146 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.table_factory: BlockBasedTable 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x556390fc1a68) 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: cache_index_and_filter_blocks: 1 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: cache_index_and_filter_blocks_with_high_priority: 0 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: pin_l0_filter_and_index_blocks_in_cache: 0 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: pin_top_level_index_and_filter: 1 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: index_type: 0 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: data_block_index_type: 0 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: index_shortening: 1 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: data_block_hash_table_util_ratio: 0.750000 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: hash_index_allow_collision: 1 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: checksum: 1 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: no_block_cache: 0 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_cache: 0x556390ff71f0 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_cache_name: BinnedLRUCache 2024-06-15T00:49:47.147 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_cache_options: 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: capacity : 536870912 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: num_shard_bits : 4 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: strict_capacity_limit : 0 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: high_pri_pool_ratio: 0.000 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_cache_compressed: (nil) 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: persistent_cache: (nil) 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_size: 4096 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_size_deviation: 10 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_restart_interval: 16 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: index_block_restart_interval: 1 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: metadata_block_size: 4096 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: partition_filters: 0 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: use_delta_encoding: 1 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: filter_policy: rocksdb.BuiltinBloomFilter 2024-06-15T00:49:47.148 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: whole_key_filtering: 1 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: verify_compression: 0 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: read_amp_bytes_per_bit: 0 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: format_version: 4 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: enable_index_compression: 1 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: block_align: 0 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.write_buffer_size: 33554432 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_write_buffer_number: 2 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression: NoCompression 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression: Disabled 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.prefix_extractor: nullptr 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.num_levels: 7 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-06-15T00:49:47.149 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.level: 32767 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.parallel_threads: 1 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bottommost_compression_opts.enabled: false 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.window_bits: -14 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.level: 32767 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.strategy: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-06-15T00:49:47.150 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.parallel_threads: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compression_opts.enabled: false 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.level0_stop_writes_trigger: 36 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.target_file_size_base: 67108864 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.target_file_size_multiplier: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_base: 268435456 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-06-15T00:49:47.151 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_compaction_bytes: 1677721600 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.arena_block_size: 4194304 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.disable_auto_compactions: 0 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-06-15T00:49:47.152 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.table_properties_collectors: 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.inplace_update_support: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.inplace_update_num_locks: 10000 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.memtable_whole_key_filtering: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.memtable_huge_page_size: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.bloom_locality: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.max_successive_merges: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.optimize_filters_for_hits: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.paranoid_file_checks: 0 2024-06-15T00:49:47.153 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.force_consistency_checks: 1 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.report_bg_io_stats: 0 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.ttl: 2592000 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.periodic_compaction_seconds: 0 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.enable_blob_files: false 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.min_blob_size: 0 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.blob_file_size: 268435456 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.blob_compression_type: NoCompression 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.enable_blob_garbage_collection: false 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: Options.blob_garbage_collection_age_cutoff: 0.250000 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/version_set.cc:4764] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi143/store.db/MANIFEST-000003 succeeded,manifest_file_number is 3, next_file_number is 5, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/version_set.cc:4779] Column family [default] (ID 0), log number is 0 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/version_set.cc:4082] Creating manifest 7 2024-06-15T00:49:47.154 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718412587027182, "job": 1, "event": "recovery_started", "wal_files": [4]} 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/db_impl/db_impl_open.cc:845] Recovering log #4 mode 2 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [table/block_based/filter_policy.cc:991] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718412587027481, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 8, "file_size": 1545, "file_checksum": "", "file_checksum_func_name": "Unknown", "table_properties": {"data_size": 523, "index_size": 31, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 1, "index_value_is_delta_encoded": 1, "filter_size": 69, "raw_key_size": 115, "raw_average_key_size": 23, "raw_value_size": 401, "raw_average_value_size": 80, "num_data_blocks": 1, "num_entries": 5, "num_deletions": 0, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1718412587, "oldest_key_time": 0, "file_creation_time": 0, "db_id": "aef6093a-7e41-4c4c-a613-c6b5c0328a3e", "db_session_id": "U6K3OHJBIDL7WC40RE16"}} 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/version_set.cc:4082] Creating manifest 9 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: EVENT_LOG_v1 {"time_micros": 1718412587027811, "job": 1, "event": "recovery_finished"} 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [file/delete_scheduler.cc:69] Deleted file /var/lib/ceph/mon/ceph-smithi143/store.db/000004.log immediately, rate_bytes_per_sec 0, total_trash_size 0 max_trash_db_ratio 0.250000 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/db_impl/db_impl_open.cc:1700] SstFileManager instance 0x55639102a700 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: DB pointer 0x556390ffa000 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/db_impl/db_impl.cc:901] ------- DUMPING STATS ------- 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: rocksdb: [db/db_impl/db_impl.cc:903] 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** DB Stats ** 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-06-15T00:49:47.155 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** Compaction Stats [default] ** 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** Compaction Stats [default] ** 2024-06-15T00:49:47.156 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Cumulative compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Interval compaction: 0.00 GB write, 0.55 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.157 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** Compaction Stats [default] ** 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: L0 1/0 1.51 KB 0.2 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Sum 1/0 1.51 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** Compaction Stats [default] ** 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 5.1 0.00 0.00 1 0.000 0 0 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Uptime(secs): 0.0 total, 0.0 interval 2024-06-15T00:49:47.158 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Flush(GB): cumulative 0.000, interval 0.000 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(GB): cumulative 0.000, interval 0.000 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(Total Files): cumulative 0, interval 0 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(L0 Files): cumulative 0, interval 0 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: AddFile(Keys): cumulative 0, interval 0 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Cumulative compaction: 0.00 GB write, 0.54 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ** File Read Latency Histogram By Level [default] ** 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143 does not exist in monmap, will attempt to join an existing cluster 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: using public_addr v2:172.21.15.143:0/0 -> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: starting mon.smithi143 rank -1 at public addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] at bind addrs [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi143 fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(???) e0 preinit fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).mds e1 new map 2024-06-15T00:49:47.159 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).mds e1 print_map 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: e1 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: enable_multiple, ever_enabled_multiple: 1,1 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: default compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: legacy client fscid: -1 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: No filesystems configured 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e0 _set_cache_ratios kv ratio 0.25 inc ratio 0.375 full ratio 0.375 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e0 register_cache_with_pcm pcm target: 2147483648 pcm max: 1020054732 pcm min: 134217728 inc_osd_cache size: 1 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e1 e1: 0 total, 0 up, 0 in 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e2 e2: 0 total, 0 up, 0 in 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e3 e3: 0 total, 0 up, 0 in 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e4 e4: 0 total, 0 up, 0 in 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e5 e5: 0 total, 0 up, 0 in 2024-06-15T00:49:47.160 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e5 crush map has features 3314932999778484224, adjusting msgr requires 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).osd e5 crush map has features 288514050185494528, adjusting msgr requires 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon alertmanager.smithi105 on smithi105 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Added host smithi143 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Health check failed: 3 mgr modules have recently crashed (RECENT_MGR_MODULE_CRASH) 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.161 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.14194 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1616808768' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1616808768' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: osdmap e4: 0 total, 0 up, 0 in 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.162 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon grafana.smithi105 on smithi105 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.14198 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi105:172.21.15.105=smithi105;smithi143:172.21.15.143=smithi143", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Saving service mon spec with placement smithi105:172.21.15.105=smithi105;smithi143:172.21.15.143=smithi143;count:2 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/205449406' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.163 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1402730181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/3304414264' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.164 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/2881269516' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon prometheus.smithi105 on smithi105 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/2030540556' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/80807275' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/654770661' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.165 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14160 172.21.15.105:0/4073552342' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mgrmap e14: smithi105.oorfym(active, since 57s) 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/2036438902' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Active manager daemon smithi105.oorfym restarted 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Activating manager daemon smithi105.oorfym 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: osdmap e5: 0 total, 0 up, 0 in 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mgrmap e15: smithi105.oorfym(active, starting, since 0.00560466s) 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi105.oorfym", "id": "smithi105.oorfym"}]: dispatch 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Manager daemon smithi105.oorfym is now available 2024-06-15T00:49:47.166 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/mirror_snapshot_schedule"}]: dispatch 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi105.oorfym/trash_purge_schedule"}]: dispatch 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mgrmap e16: smithi105.oorfym(active, since 1.00609s) 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: [15/Jun/2024:00:49:25] ENGINE Bus STARTING 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: [15/Jun/2024:00:49:25] ENGINE Serving on https://172.21.15.105:7150 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: [15/Jun/2024:00:49:25] ENGINE Bus STARTED 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: [15/Jun/2024:00:49:25] ENGINE Error in HTTPServer.serve 2024-06-15T00:49:47.167 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: Traceback (most recent call last): 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib/python3.9/site-packages/cheroot/server.py", line 1823, in serve 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: self._connections.run(self.expiration_interval) 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 203, in run 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: self._run(expiration_interval) 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 246, in _run 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: new_conn = self._from_server_socket(self.server.socket) 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib/python3.9/site-packages/cheroot/connections.py", line 300, in _from_server_socket 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: s, ssl_env = self.server.ssl_adapter.wrap(s) 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib/python3.9/site-packages/cheroot/ssl/builtin.py", line 277, in wrap 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: s = self.context.wrap_socket( 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib64/python3.9/ssl.py", line 501, in wrap_socket 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: return self.sslsocket_class._create( 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib64/python3.9/ssl.py", line 1074, in _create 2024-06-15T00:49:47.168 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: self.do_handshake() 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: File "/lib64/python3.9/ssl.py", line 1343, in do_handshake 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: self._sslobj.do_handshake() 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout: ssl.SSLZeroReturnError: TLS/SSL connection has been closed (EOF) (_ssl.c:1129) 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1504860674' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mgrmap e17: smithi105.oorfym(active, since 2s) 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd/host:smithi105", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.169 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd/host:smithi143", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi105:/etc/ceph/ceph.conf 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi143:/etc/ceph/ceph.conf 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi143:/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi105:/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:49:47.170 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Updating smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.client.admin.keyring 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/3152228922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon crash.smithi143 on smithi143 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/3900660788' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.171 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon node-exporter.smithi143 on smithi143 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/2226973720' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi143.hhtfap", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "auth get-or-create", "entity": "mgr.smithi143.hhtfap", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]': finished 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:49:47.172 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon mgr.smithi143.hhtfap on smithi143 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: Deploying daemon mon.smithi143 on smithi143 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1228870141' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 ceph-mon[35860]: mon.smithi143@-1(synchronizing).paxosservice(auth 1..6) refresh upgraded, format 0 -> 3 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 bash[35828]: 54e49738eb43ccb7b97d6e089acfa52a9966c6436c41a6ec8f1876ca9ffa08cf 2024-06-15T00:49:47.173 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:47 smithi143 systemd[1]: Started Ceph mon.smithi143 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968. 2024-06-15T00:49:47.216 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:49:52.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 -- 172.21.15.143:0/52166435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 msgr2=0x7fac68005970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:52.304 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 --2- 172.21.15.143:0/52166435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac68005970 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 crypto rx=0x7fac7c072f60 tx=0x7fac6c00b040 comp rx=0 tx=0).stop 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 -- 172.21.15.143:0/52166435 shutdown_connections 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 --2- 172.21.15.143:0/52166435 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac68005970 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 -- 172.21.15.143:0/52166435 >> 172.21.15.143:0/52166435 conn(0x7fac7c070f20 msgr2=0x7fac7c071320 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 -- 172.21.15.143:0/52166435 shutdown_connections 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.303+0000 7fac81970640 1 -- 172.21.15.143:0/52166435 wait complete. 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac81970640 1 Processor -- start 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac81970640 1 -- start start 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac81970640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac7c10d5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac81970640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac81970640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c1166d0 con 0x7fac680034f0 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac81970640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac7c116810 con 0x7fac7c10db10 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac727fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:52.305 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac727fc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:38782/0 (socket says 172.21.15.143:38782) 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.304+0000 7fac727fc640 1 -- 172.21.15.143:0/3038110873 learned_addr learned my addr 172.21.15.143:0/3038110873 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac7affd640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac7c10d5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac727fc640 1 -- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 msgr2=0x7fac7c10df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_bulk peer close file descriptor 13 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac727fc640 1 -- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 msgr2=0x7fac7c10df40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).read_until read failed 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac727fc640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac727fc640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0)._fault waiting 0.200000 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac7affd640 1 -- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 msgr2=0x7fac7c10df40 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac7affd640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:52.306 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac7affd640 1 -- 172.21.15.143:0/3038110873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac6c005040 con 0x7fac680034f0 2024-06-15T00:49:52.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac7affd640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac7c10d5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fac6c037040 tx=0x7fac6c00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:52.307 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac6c034e40 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac6c00f040 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac7c116a10 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fac6c03e5d0 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.305+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac7c116f80 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.307+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac7c0fc6c0 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.307+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fac6c0359b0 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.307+0000 7fac73fff640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fac5c0654f0 0x7fac5c0679a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.308+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fac6c0a2160 con 0x7fac680034f0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.308+0000 7fac727fc640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fac5c0654f0 0x7fac5c0679a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:52.308 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.308+0000 7fac727fc640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fac5c0654f0 0x7fac5c0679a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fac64006fd0 tx=0x7fac64008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:52.311 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.312+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fac6c071310 con 0x7fac680034f0 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mon.smithi105 calling monitor election 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: Failed to load ceph-mgr modules: k8sevents 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi143.hhtfap/crt"}]: dispatch 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mon.smithi143 calling monitor election 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mon.smithi105 is new leader, mons smithi105,smithi143 in quorum (ranks 0,1) 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: monmap e2: 2 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],smithi143=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: fsmap 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: osdmap e5: 0 total, 0 up, 0 in 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mgrmap e17: smithi105.oorfym(active, since 27s) 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mgr module k8sevents crashed in daemon mgr.smithi105.oorfym on host smithi105 at 2024-06-15T00:47:32.973957Z 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mgr module k8sevents crashed in daemon mgr.smithi105.oorfym on host smithi105 at 2024-06-15T00:47:42.459204Z 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: mgr module k8sevents crashed in daemon mgr.smithi105.oorfym on host smithi105 at 2024-06-15T00:48:19.072700Z 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: Standby manager daemon smithi143.hhtfap started 2024-06-15T00:49:52.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi143.hhtfap/key"}]: dispatch 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.388 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mon.smithi105 calling monitor election 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: Failed to load ceph-mgr modules: k8sevents 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi143.hhtfap/crt"}]: dispatch 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mon.smithi143 calling monitor election 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mon.smithi105 is new leader, mons smithi105,smithi143 in quorum (ranks 0,1) 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: monmap e2: 2 mons at {smithi105=[v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0],smithi143=[v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0]} removed_ranks: {} disallowed_leaders: {} 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: fsmap 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: osdmap e5: 0 total, 0 up, 0 in 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mgrmap e17: smithi105.oorfym(active, since 27s) 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: Health detail: HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: [WRN] RECENT_MGR_MODULE_CRASH: 3 mgr modules have recently crashed 2024-06-15T00:49:52.528 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mgr module k8sevents crashed in daemon mgr.smithi105.oorfym on host smithi105 at 2024-06-15T00:47:32.973957Z 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mgr module k8sevents crashed in daemon mgr.smithi105.oorfym on host smithi105 at 2024-06-15T00:47:42.459204Z 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: mgr module k8sevents crashed in daemon mgr.smithi105.oorfym on host smithi105 at 2024-06-15T00:48:19.072700Z 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: Standby manager daemon smithi143.hhtfap started 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/crt"}]: dispatch 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/smithi143.hhtfap/key"}]: dispatch 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.? 172.21.15.143:0/1085311608' entity='mgr.smithi143.hhtfap' cmd=[{"prefix": "config-key get", "key": "mgr/dashboard/key"}]: dispatch 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:52.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.560+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac7c04b480 con 0x7fac680034f0 2024-06-15T00:49:52.560 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.561+0000 7fac73fff640 1 -- 172.21.15.143:0/3038110873 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 2 v2) v1 ==== 95+0+1021 (secure 0 0 0) 0x7fac6c043410 con 0x7fac680034f0 2024-06-15T00:49:52.560 INFO:teuthology.orchestra.run.smithi143.stdout: 2024-06-15T00:49:52.561 INFO:teuthology.orchestra.run.smithi143.stdout:{"epoch":2,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","modified":"2024-06-15T00:49:47.055318Z","created":"2024-06-15T00:47:22.686810Z","min_mon_release":17,"min_mon_release_name":"quincy","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"tiebreaker_mon":"","removed_ranks: ":"","features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging","quincy"],"optional":[]},"mons":[{"rank":0,"name":"smithi105","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:3300","nonce":0},{"type":"v1","addr":"172.21.15.105:6789","nonce":0}]},"addr":"172.21.15.105:6789/0","public_addr":"172.21.15.105:6789/0","priority":0,"weight":0,"crush_location":"{}"},{"rank":1,"name":"smithi143","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:3300","nonce":0},{"type":"v1","addr":"172.21.15.143:6789","nonce":0}]},"addr":"172.21.15.143:6789/0","public_addr":"172.21.15.143:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0,1]} 2024-06-15T00:49:52.561 INFO:teuthology.orchestra.run.smithi143.stderr:dumped monmap epoch 2 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fac5c0654f0 msgr2=0x7fac5c0679a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fac5c0654f0 0x7fac5c0679a0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 crypto rx=0x7fac64006fd0 tx=0x7fac64008040 comp rx=0 tx=0).stop 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 msgr2=0x7fac7c10d5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac7c10d5d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 crypto rx=0x7fac6c037040 tx=0x7fac6c00b040 comp rx=0 tx=0).stop 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 shutdown_connections 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fac5c0654f0 0x7fac5c0679a0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fac7c10db10 0x7fac7c10df40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 --2- 172.21.15.143:0/3038110873 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fac680034f0 0x7fac7c10d5d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 >> 172.21.15.143:0/3038110873 conn(0x7fac7c070f20 msgr2=0x7fac7c1043c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:52.563 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 shutdown_connections 2024-06-15T00:49:52.564 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:49:52.563+0000 7fac81970640 1 -- 172.21.15.143:0/3038110873 wait complete. 2024-06-15T00:49:53.203 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:53 smithi143 ceph-mon[35860]: mgrmap e18: smithi105.oorfym(active, since 27s), standbys: smithi143.hhtfap 2024-06-15T00:49:53.203 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi143.hhtfap", "id": "smithi143.hhtfap"}]: dispatch 2024-06-15T00:49:53.203 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:49:53.203 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:53 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/3038110873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:53.203 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:53 smithi105 ceph-mon[28877]: mgrmap e18: smithi105.oorfym(active, since 27s), standbys: smithi143.hhtfap 2024-06-15T00:49:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr metadata", "who": "smithi143.hhtfap", "id": "smithi143.hhtfap"}]: dispatch 2024-06-15T00:49:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:49:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:53 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/3038110873' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-06-15T00:49:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:49:54.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:54 smithi143 ceph-mon[35860]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:54.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:54.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:54.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:54.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:49:54.525 INFO:tasks.cephadm:Generating final ceph.conf file... 2024-06-15T00:49:54.525 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph config generate-minimal-conf 2024-06-15T00:49:54.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:54 smithi105 ceph-mon[28877]: pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:54.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:54.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:54.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:54.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:49:54.790 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: Updating smithi105:/etc/ceph/ceph.conf 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: Updating smithi143:/etc/ceph/ceph.conf 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: Updating smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: Updating smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:49:55.213 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:49:55.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:55.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.214 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: Updating smithi105:/etc/ceph/ceph.conf 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: Updating smithi143:/etc/ceph/ceph.conf 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: Updating smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: Updating smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/config/ceph.conf 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:49:55.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:49:55.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:55.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:55.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:49:56.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:56 smithi105 ceph-mon[28877]: Reconfiguring mon.smithi105 (unknown last config time)... 2024-06-15T00:49:56.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:56 smithi105 ceph-mon[28877]: Reconfiguring daemon mon.smithi105 on smithi105 2024-06-15T00:49:56.413 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:56 smithi105 ceph-mon[28877]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 -- 172.21.15.105:0/1162737255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 msgr2=0x7fa968102310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 --2- 172.21.15.105:0/1162737255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968102310 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 crypto rx=0x7fa960004850 tx=0x7fa960009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 -- 172.21.15.105:0/1162737255 shutdown_connections 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 --2- 172.21.15.105:0/1162737255 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968102310 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 -- 172.21.15.105:0/1162737255 >> 172.21.15.105:0/1162737255 conn(0x7fa9680f7b10 msgr2=0x7fa9680f9f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 -- 172.21.15.105:0/1162737255 shutdown_connections 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.489+0000 7fa9703b2640 1 -- 172.21.15.105:0/1162737255 wait complete. 2024-06-15T00:49:56.572 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 Processor -- start 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 -- start start 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968112610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968112b50 0x7fa96810c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa968113160 con 0x7fa968101f40 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa96810caf0 con 0x7fa968112b50 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968112610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968112610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60444/0 (socket says 172.21.15.105:60444) 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 -- 172.21.15.105:0/3514121386 learned_addr learned my addr 172.21.15.105:0/3514121386 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 -- 172.21.15.105:0/3514121386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968112b50 msgr2=0x7fa96810c5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968112b50 0x7fa96810c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 -- 172.21.15.105:0/3514121386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa960005040 con 0x7fa968101f40 2024-06-15T00:49:56.573 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa96e14e640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968112610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa96002d040 tx=0x7fa960009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa960038070 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa960032b60 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa96810cd70 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.490+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa96003db10 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.491+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa96810d230 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.491+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa96810d4a0 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.492+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fa960030020 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.492+0000 7fa966ffd640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa950065500 0x7fa9500679b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.492+0000 7fa96d94d640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa950065500 0x7fa9500679b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.492+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7fa9600a2120 con 0x7fa968101f40 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.493+0000 7fa96d94d640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa950065500 0x7fa9500679b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00acf0 tx=0x7fa95c009370 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:49:56.574 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.497+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa9600714b0 con 0x7fa968101f40 2024-06-15T00:49:56.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:56 smithi143 ceph-mon[35860]: Reconfiguring mon.smithi105 (unknown last config time)... 2024-06-15T00:49:56.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:56 smithi143 ceph-mon[35860]: Reconfiguring daemon mon.smithi105 on smithi105 2024-06-15T00:49:56.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:56 smithi143 ceph-mon[35860]: pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:56.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.662+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7fa96804b480 con 0x7fa968101f40 2024-06-15T00:49:56.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.662+0000 7fa966ffd640 1 -- 172.21.15.105:0/3514121386 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v10) v1 ==== 76+0+227 (secure 0 0 0) 0x7fa9600a18d0 con 0x7fa968101f40 2024-06-15T00:49:56.663 INFO:teuthology.orchestra.run.smithi105.stdout:# minimal ceph.conf for bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:49:56.663 INFO:teuthology.orchestra.run.smithi105.stdout:[global] 2024-06-15T00:49:56.663 INFO:teuthology.orchestra.run.smithi105.stdout: fsid = bd0fff6c-2ab0-11ef-bc9d-c7b262605968 2024-06-15T00:49:56.663 INFO:teuthology.orchestra.run.smithi105.stdout: mon_host = [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] 2024-06-15T00:49:56.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.664+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa950065500 msgr2=0x7fa9500679b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.664+0000 7fa9703b2640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa950065500 0x7fa9500679b0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 crypto rx=0x7fa95c00acf0 tx=0x7fa95c009370 comp rx=0 tx=0).stop 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.664+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 msgr2=0x7fa968112610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.664+0000 7fa9703b2640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968112610 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 crypto rx=0x7fa96002d040 tx=0x7fa960009da0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.664+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 shutdown_connections 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.665+0000 7fa9703b2640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa950065500 0x7fa9500679b0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.665+0000 7fa9703b2640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa968112b50 0x7fa96810c5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.665+0000 7fa9703b2640 1 --2- 172.21.15.105:0/3514121386 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa968101f40 0x7fa968112610 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.665+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 >> 172.21.15.105:0/3514121386 conn(0x7fa9680f7b10 msgr2=0x7fa9680f96c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.665+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 shutdown_connections 2024-06-15T00:49:56.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:49:56.665+0000 7fa9703b2640 1 -- 172.21.15.105:0/3514121386 wait complete. 2024-06-15T00:49:57.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:57 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3514121386' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:57.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:57 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3514121386' entity='client.admin' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:57.644 INFO:tasks.cephadm:Distributing (final) config and client.admin keyring... 2024-06-15T00:49:57.644 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:49:57.644 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.conf 2024-06-15T00:49:57.673 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:49:57.673 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:49:57.738 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:49:57.738 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.conf 2024-06-15T00:49:57.771 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:49:57.771 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.admin.keyring 2024-06-15T00:49:57.841 INFO:tasks.cephadm:Deploying OSDs... 2024-06-15T00:49:57.841 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:49:57.841 DEBUG:teuthology.orchestra.run.smithi105:> dd if=/scratch_devs of=/dev/stdout 2024-06-15T00:49:57.857 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-15T00:49:57.857 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_1 2024-06-15T00:49:57.913 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 675 Links: 1 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-06-15 00:49:18.619922122 +0000 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-06-15 00:43:39.877076857 +0000 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-06-15 00:43:39.877076857 +0000 2024-06-15T00:49:57.914 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-06-15 00:43:39.877076857 +0000 2024-06-15T00:49:57.914 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-15T00:49:57.977 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-06-15T00:49:57.977 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-06-15T00:49:57.977 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000217997 s, 2.3 MB/s 2024-06-15T00:49:57.979 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-15T00:49:58.034 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_2 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 695 Links: 1 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-06-15 00:49:18.619922122 +0000 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-06-15 00:43:40.219071123 +0000 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-06-15 00:43:40.219071123 +0000 2024-06-15T00:49:58.090 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-06-15 00:43:40.219071123 +0000 2024-06-15T00:49:58.091 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-15T00:49:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-06-15T00:49:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-06-15T00:49:58.153 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000178073 s, 2.9 MB/s 2024-06-15T00:49:58.155 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-15T00:49:58.213 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_3 2024-06-15T00:49:58.268 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 712 Links: 1 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-06-15 00:49:18.620922105 +0000 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-06-15 00:43:40.564065338 +0000 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-06-15 00:43:40.564065338 +0000 2024-06-15T00:49:58.269 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-06-15 00:43:40.564065338 +0000 2024-06-15T00:49:58.269 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-15T00:49:58.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:58 smithi105 ceph-mon[28877]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:58.298 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-06-15T00:49:58.299 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-06-15T00:49:58.299 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.000213933 s, 2.4 MB/s 2024-06-15T00:49:58.300 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-15T00:49:58.357 DEBUG:teuthology.orchestra.run.smithi105:> stat /dev/vg_nvme/lv_4 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout:Device: 5h/5d Inode: 731 Links: 1 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout:Access: 2024-06-15 00:49:18.620922105 +0000 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout:Modify: 2024-06-15 00:43:40.902059670 +0000 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout:Change: 2024-06-15 00:43:40.902059670 +0000 2024-06-15T00:49:58.413 INFO:teuthology.orchestra.run.smithi105.stdout: Birth: 2024-06-15 00:43:40.902059670 +0000 2024-06-15T00:49:58.414 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-15T00:49:58.477 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records in 2024-06-15T00:49:58.478 INFO:teuthology.orchestra.run.smithi105.stderr:1+0 records out 2024-06-15T00:49:58.478 INFO:teuthology.orchestra.run.smithi105.stderr:512 bytes copied, 0.00020169 s, 2.5 MB/s 2024-06-15T00:49:58.479 DEBUG:teuthology.orchestra.run.smithi105:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-15T00:49:58.536 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:49:58.536 DEBUG:teuthology.orchestra.run.smithi143:> dd if=/scratch_devs of=/dev/stdout 2024-06-15T00:49:58.556 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2024-06-15T00:49:58.556 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_1 2024-06-15T00:49:58.614 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 672 Links: 1 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-06-15 00:49:46.300716673 +0000 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-06-15 00:43:46.390487413 +0000 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-06-15 00:43:46.390487413 +0000 2024-06-15T00:49:58.615 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-06-15 00:43:46.390487413 +0000 2024-06-15T00:49:58.615 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 2024-06-15T00:49:58.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:58 smithi143 ceph-mon[35860]: pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:49:58.661 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-06-15T00:49:58.662 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-06-15T00:49:58.662 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.00029395 s, 1.7 MB/s 2024-06-15T00:49:58.663 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_1 2024-06-15T00:49:58.722 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_2 2024-06-15T00:49:58.779 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2024-06-15T00:49:58.779 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:58.779 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 690 Links: 1 2024-06-15T00:49:58.779 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:58.779 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:58.780 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-06-15 00:49:46.300716673 +0000 2024-06-15T00:49:58.780 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-06-15 00:43:46.727478341 +0000 2024-06-15T00:49:58.780 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-06-15 00:43:46.727478341 +0000 2024-06-15T00:49:58.780 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-06-15 00:43:46.727478341 +0000 2024-06-15T00:49:58.780 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 2024-06-15T00:49:58.845 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-06-15T00:49:58.845 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-06-15T00:49:58.845 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000242567 s, 2.1 MB/s 2024-06-15T00:49:58.846 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_2 2024-06-15T00:49:58.903 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_3 2024-06-15T00:49:58.959 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 709 Links: 1 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-06-15 00:49:46.301716647 +0000 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-06-15 00:43:47.069469136 +0000 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-06-15 00:43:47.069469136 +0000 2024-06-15T00:49:58.960 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-06-15 00:43:47.069469136 +0000 2024-06-15T00:49:58.960 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 2024-06-15T00:49:59.023 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-06-15T00:49:59.024 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-06-15T00:49:59.024 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000197562 s, 2.6 MB/s 2024-06-15T00:49:59.025 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_3 2024-06-15T00:49:59.082 DEBUG:teuthology.orchestra.run.smithi143:> stat /dev/vg_nvme/lv_4 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout:Device: 5h/5d Inode: 726 Links: 1 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout:Context: system_u:object_r:device_t:s0 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout:Access: 2024-06-15 00:49:46.301716647 +0000 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout:Modify: 2024-06-15 00:43:47.408460010 +0000 2024-06-15T00:49:59.138 INFO:teuthology.orchestra.run.smithi143.stdout:Change: 2024-06-15 00:43:47.408460010 +0000 2024-06-15T00:49:59.139 INFO:teuthology.orchestra.run.smithi143.stdout: Birth: 2024-06-15 00:43:47.408460010 +0000 2024-06-15T00:49:59.139 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 2024-06-15T00:49:59.202 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records in 2024-06-15T00:49:59.202 INFO:teuthology.orchestra.run.smithi143.stderr:1+0 records out 2024-06-15T00:49:59.202 INFO:teuthology.orchestra.run.smithi143.stderr:512 bytes copied, 0.000164566 s, 3.1 MB/s 2024-06-15T00:49:59.203 DEBUG:teuthology.orchestra.run.smithi143:> ! mount | grep -v devtmpfs | grep -q /dev/vg_nvme/lv_4 2024-06-15T00:49:59.259 INFO:tasks.cephadm:Deploying osd.0 on smithi105 with /dev/vg_nvme/lv_4... 2024-06-15T00:49:59.260 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-15T00:49:59.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:59.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:59.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: Reconfiguring mgr.smithi105.oorfym (unknown last config time)... 2024-06-15T00:49:59.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.oorfym", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:49:59.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:49:59.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:59.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:49:59 smithi143 ceph-mon[35860]: Reconfiguring daemon mgr.smithi105.oorfym on smithi105 2024-06-15T00:49:59.522 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:59.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:49:59.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: Reconfiguring mgr.smithi105.oorfym (unknown last config time)... 2024-06-15T00:49:59.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi105.oorfym", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:49:59.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:49:59.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:49:59.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:49:59 smithi105 ceph-mon[28877]: Reconfiguring daemon mgr.smithi105.oorfym on smithi105 2024-06-15T00:49:59.593 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:50:00.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:00 smithi105 ceph-mon[28877]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:00.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:00 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:00.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:00 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:00.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:00 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:50:00.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:00 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:00.477 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:00 smithi105 ceph-mon[28877]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:50:00.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:00 smithi143 ceph-mon[35860]: pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:00.654 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:00 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:00.654 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:00 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:00.654 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:00 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi105", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:50:00.654 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:00 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:00.654 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:00 smithi143 ceph-mon[35860]: overall HEALTH_WARN 3 mgr modules have recently crashed 2024-06-15T00:50:01.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:01 smithi105 ceph-mon[28877]: Reconfiguring crash.smithi105 (monmap changed)... 2024-06-15T00:50:01.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:01 smithi105 ceph-mon[28877]: Reconfiguring daemon crash.smithi105 on smithi105 2024-06-15T00:50:01.537 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:01 smithi143 ceph-mon[35860]: Reconfiguring crash.smithi105 (monmap changed)... 2024-06-15T00:50:01.537 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:01 smithi143 ceph-mon[35860]: Reconfiguring daemon crash.smithi105 on smithi105 2024-06-15T00:50:02.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:02 smithi105 ceph-mon[28877]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:02.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:02 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:02.290 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:02 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:02.330 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-15T00:50:02.415 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-15T00:50:02.428 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-06-15T00:50:02.428 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-06-15T00:50:02.428 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.011392 s, 920 MB/s 2024-06-15T00:50:02.428 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-06-15T00:50:02.554 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:02 smithi143 ceph-mon[35860]: pgmap v11: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:02.554 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:02 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:02.554 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:02 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:03.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:03 smithi105 ceph-mon[28877]: Reconfiguring alertmanager.smithi105 (dependencies changed)... 2024-06-15T00:50:03.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:03 smithi105 ceph-mon[28877]: Reconfiguring daemon alertmanager.smithi105 on smithi105 2024-06-15T00:50:03.617 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_4 2024-06-15T00:50:03.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:03 smithi143 ceph-mon[35860]: Reconfiguring alertmanager.smithi105 (dependencies changed)... 2024-06-15T00:50:03.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:03 smithi143 ceph-mon[35860]: Reconfiguring daemon alertmanager.smithi105 on smithi105 2024-06-15T00:50:04.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:04 smithi105 ceph-mon[28877]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:04.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:04 smithi143 ceph-mon[35860]: pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:04.883 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:50:06.440 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:06 smithi105 ceph-mon[28877]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:06.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:06 smithi143 ceph-mon[35860]: pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:07.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 -- 172.21.15.105:0/1276953401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0659a0 msgr2=0x7f71fc0feea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:50:07.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 --2- 172.21.15.105:0/1276953401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0659a0 0x7f71fc0feea0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 crypto rx=0x7f71ec0048d0 tx=0x7f71ec009da0 comp rx=0 tx=0).stop 2024-06-15T00:50:07.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 -- 172.21.15.105:0/1276953401 shutdown_connections 2024-06-15T00:50:07.741 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 --2- 172.21.15.105:0/1276953401 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc1017c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 --2- 172.21.15.105:0/1276953401 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0659a0 0x7f71fc0feea0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 -- 172.21.15.105:0/1276953401 >> 172.21.15.105:0/1276953401 conn(0x7f71fc0f6970 msgr2=0x7f71fc0f8d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 -- 172.21.15.105:0/1276953401 shutdown_connections 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 -- 172.21.15.105:0/1276953401 wait complete. 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 Processor -- start 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.705+0000 7f7202a6f640 1 -- start start 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f7202a6f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71fc0659a0 0x7f71fc10cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f7202a6f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc10d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f7202a6f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc10e170 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f7202a6f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71fc10e2e0 con 0x7f71fc0659a0 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc10d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc10d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42032/0 (socket says 172.21.15.105:42032) 2024-06-15T00:50:07.742 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 -- 172.21.15.105:0/3032620476 learned_addr learned my addr 172.21.15.105:0/3032620476 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 -- 172.21.15.105:0/3032620476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71fc0659a0 msgr2=0x7f71fc10cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71fc0659a0 0x7f71fc10cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 -- 172.21.15.105:0/3032620476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71ec005040 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71fbfff640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc10d490 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f71f000b790 tx=0x7f71f00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71f000c700 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f71f000cd10 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f71f0013370 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.706+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71fc10e5c0 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.707+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71fc10eae0 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.708+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71fc0fa5f0 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.708+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7f71f0012070 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.743 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.708+0000 7f71f97fa640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f71e40653a0 0x7f71e4067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:50:07.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.708+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1366+0+0 (secure 0 0 0) 0x7f71f0081250 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.708+0000 7f720080b640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f71e40653a0 0x7f71e4067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:50:07.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.709+0000 7f720080b640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f71e40653a0 0x7f71e4067850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71ec003080 tx=0x7f71ec009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:50:07.744 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.712+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f71f0050520 con 0x7f71fc0ff3e0 2024-06-15T00:50:07.899 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:07.899+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7f71fc10ee20 con 0x7f71e40653a0 2024-06-15T00:50:08.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:08 smithi105 ceph-mon[28877]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:08.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:08 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:50:08.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:08 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:50:08.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:08 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:08.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:08 smithi143 ceph-mon[35860]: pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:08.638 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:08 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:50:08.638 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:08 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:50:08.638 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:08 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:09.448 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:09 smithi105 ceph-mon[28877]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:50:09.448 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:09 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:09.448 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:09 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:09.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:09 smithi143 ceph-mon[35860]: from='client.14246 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:50:09.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:09 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:09.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:09 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:10.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:10 smithi105 ceph-mon[28877]: Reconfiguring grafana.smithi105 (dependencies changed)... 2024-06-15T00:50:10.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:10 smithi105 ceph-mon[28877]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:10.374 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:10 smithi105 ceph-mon[28877]: Reconfiguring daemon grafana.smithi105 on smithi105 2024-06-15T00:50:10.375 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:10 smithi143 ceph-mon[35860]: Reconfiguring grafana.smithi105 (dependencies changed)... 2024-06-15T00:50:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:10 smithi143 ceph-mon[35860]: pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:10 smithi143 ceph-mon[35860]: Reconfiguring daemon grafana.smithi105 on smithi105 2024-06-15T00:50:10.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:12.269 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:12 smithi105 ceph-mon[28877]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:12.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:12 smithi143 ceph-mon[35860]: pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:14.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:14 smithi105 ceph-mon[28877]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:14.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:14 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:14.378 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:14 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:14.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:14 smithi143 ceph-mon[35860]: pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:14.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:14 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:14.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:14 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:15.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: Reconfiguring prometheus.smithi105 (dependencies changed)... 2024-06-15T00:50:15.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: Reconfiguring daemon prometheus.smithi105 on smithi105 2024-06-15T00:50:15.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3832212231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e146f885-f9f1-4f23-b08d-e3efaf314c9b"}]: dispatch 2024-06-15T00:50:15.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3832212231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e146f885-f9f1-4f23-b08d-e3efaf314c9b"}]': finished 2024-06-15T00:50:15.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: osdmap e6: 1 total, 0 up, 1 in 2024-06-15T00:50:15.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:15.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:15 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3246167396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:50:15.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: Reconfiguring prometheus.smithi105 (dependencies changed)... 2024-06-15T00:50:15.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: Reconfiguring daemon prometheus.smithi105 on smithi105 2024-06-15T00:50:15.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3832212231' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "e146f885-f9f1-4f23-b08d-e3efaf314c9b"}]: dispatch 2024-06-15T00:50:15.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3832212231' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "e146f885-f9f1-4f23-b08d-e3efaf314c9b"}]': finished 2024-06-15T00:50:15.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: osdmap e6: 1 total, 0 up, 1 in 2024-06-15T00:50:15.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:15.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:15 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3246167396' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:50:16.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:16 smithi143 ceph-mon[35860]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:16.678 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:16 smithi105 ceph-mon[28877]: pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:18.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:18 smithi105 ceph-mon[28877]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:18.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:18 smithi143 ceph-mon[35860]: pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:20.403 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:20 smithi105 ceph-mon[28877]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:20.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:20 smithi143 ceph-mon[35860]: pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:22.507 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:22 smithi105 ceph-mon[28877]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:22.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:22 smithi143 ceph-mon[35860]: pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:24.507 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:24 smithi105 ceph-mon[28877]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:24.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:24 smithi143 ceph-mon[35860]: pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:25.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:25.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:25.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:50:25.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:25.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:25.672 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:25.672 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:25.672 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi143", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-06-15T00:50:25.672 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:25.672 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:26.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: Reconfiguring crash.smithi143 (monmap changed)... 2024-06-15T00:50:26.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: Reconfiguring daemon crash.smithi143 on smithi143 2024-06-15T00:50:26.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:26.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:26.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:26.425 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi143.hhtfap", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:50:26.426 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:50:26.426 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:26.426 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-15T00:50:26.426 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:26.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: Reconfiguring crash.smithi143 (monmap changed)... 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: Reconfiguring daemon crash.smithi143 on smithi143 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get-or-create", "entity": "mgr.smithi143.hhtfap", "caps": ["mon", "profile mgr", "osd", "allow *", "mds", "allow *"]}]: dispatch 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mgr services"}]: dispatch 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.0"}]: dispatch 2024-06-15T00:50:26.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: Reconfiguring mgr.smithi143.hhtfap (monmap changed)... 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: Reconfiguring daemon mgr.smithi143.hhtfap on smithi143 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: Deploying daemon osd.0 on smithi105 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:50:27.528 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:50:27.529 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: Reconfiguring mgr.smithi143.hhtfap (monmap changed)... 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: Reconfiguring daemon mgr.smithi143.hhtfap on smithi143 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: Deploying daemon osd.0 on smithi105 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "mon."}]: dispatch 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config get", "who": "mon", "key": "public_network"}]: dispatch 2024-06-15T00:50:27.600 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:28.525 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:28.525 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: Reconfiguring mon.smithi143 (monmap changed)... 2024-06-15T00:50:28.525 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: Reconfiguring daemon mon.smithi143 on smithi143 2024-06-15T00:50:28.525 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.525 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.525 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:50:28.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:50:28.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:50:28.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:28.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: Reconfiguring mon.smithi143 (monmap changed)... 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: Reconfiguring daemon mon.smithi143 on smithi143 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-alertmanager-api-host"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-alertmanager-api-host", "value": "http://smithi105.front.sepia.ceph.com:9093"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-grafana-api-url"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:50:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-url", "value": "https://smithi105.front.sepia.ceph.com:3000"}]: dispatch 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard get-prometheus-api-host"}]: dispatch 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-prometheus-api-host", "value": "http://smithi105.front.sepia.ceph.com:9095"}]: dispatch 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:28.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:50:29.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:29 smithi105 ceph-mon[28877]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:29.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:29 smithi143 ceph-mon[35860]: pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:31.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:30 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:31.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:30 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:31.293 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:30 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:31.293 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:30 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:32.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:32 smithi105 ceph-mon[28877]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:32.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:32 smithi143 ceph-mon[35860]: pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:34.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:34 smithi105 ceph-mon[28877]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:34 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:34 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:34 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:34 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:34 smithi143 ceph-mon[35860]: pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:34 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:34 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:34 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:34.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:34 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:35.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:35 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:35.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:35 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:50:35.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:35 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:35.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:35 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:50:35.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:35 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:35.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:35 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:50:35.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:35 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:35.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:35 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:50:36.199 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:36 smithi105 ceph-mon[28877]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:36.199 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:36.199 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:36.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:36 smithi143 ceph-mon[35860]: pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:36.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:36 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:36.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:36 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:36.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.820+0000 7f71f97fa640 1 -- 172.21.15.105:0/3032620476 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f71fc10ee20 con 0x7f71e40653a0 2024-06-15T00:50:36.820 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 0 on host 'smithi105' 2024-06-15T00:50:36.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f71e40653a0 msgr2=0x7f71e4067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:50:36.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f71e40653a0 0x7f71e4067850 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 crypto rx=0x7f71ec003080 tx=0x7f71ec009da0 comp rx=0 tx=0).stop 2024-06-15T00:50:36.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 msgr2=0x7f71fc10d490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc10d490 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 crypto rx=0x7f71f000b790 tx=0x7f71f00091d0 comp rx=0 tx=0).stop 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 shutdown_connections 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f71e40653a0 0x7f71e4067850 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f71fc0ff3e0 0x7f71fc10d490 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 --2- 172.21.15.105:0/3032620476 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f71fc0659a0 0x7f71fc10cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 >> 172.21.15.105:0/3032620476 conn(0x7f71fc0f6970 msgr2=0x7f71fc0f6d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 shutdown_connections 2024-06-15T00:50:36.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:36.822+0000 7f7202a6f640 1 -- 172.21.15.105:0/3032620476 wait complete. 2024-06-15T00:50:37.789 DEBUG:teuthology.orchestra.run.smithi105:osd.0> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.0.service 2024-06-15T00:50:37.791 INFO:tasks.cephadm:Deploying osd.1 on smithi105 with /dev/vg_nvme/lv_3... 2024-06-15T00:50:37.792 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-15T00:50:38.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:37 smithi105 ceph-mon[28877]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:38.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:37 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:38.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:37 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:38.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:37 smithi105 ceph-mon[28877]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-15T00:50:38.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:37 smithi143 ceph-mon[35860]: pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:38.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:37 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:38.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:37 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:38.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:37 smithi143 ceph-mon[35860]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]: dispatch 2024-06-15T00:50:38.641 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:50:39.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:39 smithi105 ceph-mon[28877]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-15T00:50:39.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:39 smithi105 ceph-mon[28877]: osdmap e7: 1 total, 0 up, 1 in 2024-06-15T00:50:39.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:39.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:39 smithi105 ceph-mon[28877]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-06-15T00:50:39.527 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:50:39 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0[46846]: 2024-06-15T00:50:39.187+0000 7f5ba1edf640 -1 osd.0 0 waiting for initial osdmap 2024-06-15T00:50:39.527 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:50:39 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0[46846]: 2024-06-15T00:50:39.190+0000 7f5b9acf0640 -1 osd.0 8 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:50:39.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:39 smithi143 ceph-mon[35860]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["0"]}]': finished 2024-06-15T00:50:39.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:39 smithi143 ceph-mon[35860]: osdmap e7: 1 total, 0 up, 1 in 2024-06-15T00:50:39.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:39.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:39 smithi143 ceph-mon[35860]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd=[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-06-15T00:50:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:40 smithi105 ceph-mon[28877]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:40 smithi105 ceph-mon[28877]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-06-15T00:50:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:40 smithi105 ceph-mon[28877]: osdmap e8: 1 total, 0 up, 1 in 2024-06-15T00:50:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:40.416 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-15T00:50:40.508 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-15T00:50:40.522 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-06-15T00:50:40.522 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-06-15T00:50:40.522 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0124673 s, 841 MB/s 2024-06-15T00:50:40.522 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-06-15T00:50:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:40 smithi143 ceph-mon[35860]: pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:40 smithi143 ceph-mon[35860]: from='osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752]' entity='osd.0' cmd='[{"prefix": "osd crush create-or-move", "id": 0, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-06-15T00:50:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:40 smithi143 ceph-mon[35860]: osdmap e8: 1 total, 0 up, 1 in 2024-06-15T00:50:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:40.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:41.437 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:41 smithi105 ceph-mon[28877]: purged_snaps scrub starts 2024-06-15T00:50:41.437 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:41 smithi105 ceph-mon[28877]: purged_snaps scrub ok 2024-06-15T00:50:41.438 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:41 smithi105 ceph-mon[28877]: osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] boot 2024-06-15T00:50:41.438 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:41 smithi105 ceph-mon[28877]: osdmap e9: 1 total, 1 up, 1 in 2024-06-15T00:50:41.438 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:41 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:41.461 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_3 2024-06-15T00:50:41.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:41 smithi143 ceph-mon[35860]: purged_snaps scrub starts 2024-06-15T00:50:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:41 smithi143 ceph-mon[35860]: purged_snaps scrub ok 2024-06-15T00:50:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:41 smithi143 ceph-mon[35860]: osd.0 [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] boot 2024-06-15T00:50:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:41 smithi143 ceph-mon[35860]: osdmap e9: 1 total, 1 up, 1 in 2024-06-15T00:50:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 0}]: dispatch 2024-06-15T00:50:42.236 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:50:42.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:42 smithi105 ceph-mon[28877]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:42.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:42 smithi105 ceph-mon[28877]: osdmap e10: 1 total, 1 up, 1 in 2024-06-15T00:50:42.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:42 smithi143 ceph-mon[35860]: pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-06-15T00:50:42.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:42 smithi143 ceph-mon[35860]: osdmap e10: 1 total, 1 up, 1 in 2024-06-15T00:50:43.973 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:43 smithi105 ceph-mon[28877]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:43.973 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:43 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:43.973 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:43 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:44.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:43 smithi143 ceph-mon[35860]: pgmap v37: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:44.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:43 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:44.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:43 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:45.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 -- 172.21.15.105:0/600939877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fc2b0 msgr2=0x7fdcf00fc6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:50:45.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 --2- 172.21.15.105:0/600939877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fc2b0 0x7fdcf00fc6c0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0048d0 tx=0x7fdcec009da0 comp rx=0 tx=0).stop 2024-06-15T00:50:45.113 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 -- 172.21.15.105:0/600939877 shutdown_connections 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 --2- 172.21.15.105:0/600939877 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf00fd580 0x7fdcf00fd9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 --2- 172.21.15.105:0/600939877 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fc2b0 0x7fdcf00fc6c0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 -- 172.21.15.105:0/600939877 >> 172.21.15.105:0/600939877 conn(0x7fdcf00f7a90 msgr2=0x7fdcf00f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 -- 172.21.15.105:0/600939877 shutdown_connections 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.674+0000 7fdcf813d640 1 -- 172.21.15.105:0/600939877 wait complete. 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf813d640 1 Processor -- start 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf813d640 1 -- start start 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf813d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf00fc2b0 0x7fdcf010cee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf813d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 0x7fdcf010d420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf813d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf010e1b0 con 0x7fdcf00fd580 2024-06-15T00:50:45.114 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf813d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcf010e320 con 0x7fdcf00fc2b0 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf56d8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 0x7fdcf010d420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf56d8640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 0x7fdcf010d420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40718/0 (socket says 172.21.15.105:40718) 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.675+0000 7fdcf56d8640 1 -- 172.21.15.105:0/3917816373 learned_addr learned my addr 172.21.15.105:0/3917816373 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf5ed9640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf00fc2b0 0x7fdcf010cee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf56d8640 1 -- 172.21.15.105:0/3917816373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf00fc2b0 msgr2=0x7fdcf010cee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf56d8640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf00fc2b0 0x7fdcf010cee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf56d8640 1 -- 172.21.15.105:0/3917816373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec005040 con 0x7fdcf00fd580 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf56d8640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 0x7fdcf010d420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdce000b790 tx=0x7fdce00091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdce000c700 con 0x7fdcf00fd580 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdce000cd10 con 0x7fdcf00fd580 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcf010e600 con 0x7fdcf00fd580 2024-06-15T00:50:45.115 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdce0013370 con 0x7fdcf00fd580 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.676+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcf010eb20 con 0x7fdcf00fd580 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.677+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcf00fb710 con 0x7fdcf00fd580 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.677+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fdce0012070 con 0x7fdcf00fd580 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.678+0000 7fdce67fc640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdcdc0653a0 0x7fdcdc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.678+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(10..10 src has 1..10) v4 ==== 1978+0+0 (secure 0 0 0) 0x7fdce00815f0 con 0x7fdcf00fd580 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.678+0000 7fdcf5ed9640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdcdc0653a0 0x7fdcdc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.678+0000 7fdcf5ed9640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdcdc0653a0 0x7fdcdc067850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007ac0 tx=0x7fdcec035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.682+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdce00505e0 con 0x7fdcf00fd580 2024-06-15T00:50:45.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:50:44.869+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7fdcf010ee10 con 0x7fdcdc0653a0 2024-06-15T00:50:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:50:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:50:45.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:44 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:45.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:44 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:50:45.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:44 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:50:45.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:44 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:45.965 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:45 smithi105 ceph-mon[28877]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:45.965 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:45 smithi105 ceph-mon[28877]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:50:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:45 smithi143 ceph-mon[35860]: pgmap v38: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:45 smithi143 ceph-mon[35860]: from='client.14264 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:50:48.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: pgmap v39: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:48.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:50:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:50:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:50:49.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:48 smithi105 ceph-mon[28877]: Detected new or changed devices on smithi105 2024-06-15T00:50:49.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:48 smithi105 ceph-mon[28877]: Adjusting osd_memory_target on smithi105 to 12864M 2024-06-15T00:50:49.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:48 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3583114794' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e67024-91f3-4938-9f5f-df5ef15c20e2"}]: dispatch 2024-06-15T00:50:49.192 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:48 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3583114794' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4e67024-91f3-4938-9f5f-df5ef15c20e2"}]': finished 2024-06-15T00:50:49.193 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:48 smithi105 ceph-mon[28877]: osdmap e11: 2 total, 1 up, 2 in 2024-06-15T00:50:49.193 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:48 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:50:49.226 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:48 smithi143 ceph-mon[35860]: Detected new or changed devices on smithi105 2024-06-15T00:50:49.226 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:48 smithi143 ceph-mon[35860]: Adjusting osd_memory_target on smithi105 to 12864M 2024-06-15T00:50:49.226 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:48 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3583114794' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "f4e67024-91f3-4938-9f5f-df5ef15c20e2"}]: dispatch 2024-06-15T00:50:49.226 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:48 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3583114794' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "f4e67024-91f3-4938-9f5f-df5ef15c20e2"}]': finished 2024-06-15T00:50:49.226 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:48 smithi143 ceph-mon[35860]: osdmap e11: 2 total, 1 up, 2 in 2024-06-15T00:50:49.226 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:48 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:50:50.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:49 smithi105 ceph-mon[28877]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:50.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:49 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2487536292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:50:50.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:49 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:50.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:49 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:50.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:49 smithi143 ceph-mon[35860]: pgmap v40: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:50.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:49 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2487536292' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:50:50.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:49 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:50.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:49 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:52.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:51 smithi105 ceph-mon[28877]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:52.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:51 smithi143 ceph-mon[35860]: pgmap v42: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:53.364 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:53.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:53.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:53.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:50:53.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:53.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:50:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:50:54.474 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:54 smithi105 ceph-mon[28877]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:54.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:54 smithi143 ceph-mon[35860]: pgmap v43: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:55.464 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:55.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:50:56.618 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:56 smithi105 ceph-mon[28877]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:56.618 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:56 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-15T00:50:56.618 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:56 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:56.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:56 smithi143 ceph-mon[35860]: pgmap v44: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:56.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:56 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.1"}]: dispatch 2024-06-15T00:50:56.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:56 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:50:57.571 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:57 smithi105 ceph-mon[28877]: Deploying daemon osd.1 on smithi105 2024-06-15T00:50:57.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:57 smithi143 ceph-mon[35860]: Deploying daemon osd.1 on smithi105 2024-06-15T00:50:58.611 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:50:58 smithi105 ceph-mon[28877]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:50:58.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:50:58 smithi143 ceph-mon[35860]: pgmap v45: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:00.619 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:00 smithi105 ceph-mon[28877]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:00.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:00 smithi143 ceph-mon[35860]: pgmap v46: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:01.588 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:01.588 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:01.589 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:01.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:01.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:01.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:02.603 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:02 smithi105 ceph-mon[28877]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:02.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:02 smithi143 ceph-mon[35860]: pgmap v47: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.172+0000 7fdce67fc640 1 -- 172.21.15.105:0/3917816373 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fdcf010ee10 con 0x7fdcdc0653a0 2024-06-15T00:51:04.173 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 1 on host 'smithi105' 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdcdc0653a0 msgr2=0x7fdcdc067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdcdc0653a0 0x7fdcdc067850 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fdcec007ac0 tx=0x7fdcec035040 comp rx=0 tx=0).stop 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 msgr2=0x7fdcf010d420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 0x7fdcf010d420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 crypto rx=0x7fdce000b790 tx=0x7fdce00091d0 comp rx=0 tx=0).stop 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 shutdown_connections 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdcdc0653a0 0x7fdcdc067850 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:04.174 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcf00fd580 0x7fdcf010d420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:04.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 --2- 172.21.15.105:0/3917816373 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcf00fc2b0 0x7fdcf010cee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:04.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 >> 172.21.15.105:0/3917816373 conn(0x7fdcf00f7a90 msgr2=0x7fdcf00f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:51:04.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 shutdown_connections 2024-06-15T00:51:04.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:04.174+0000 7fdcf813d640 1 -- 172.21.15.105:0/3917816373 wait complete. 2024-06-15T00:51:04.481 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:04 smithi105 ceph-mon[28877]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:04.481 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:04 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:04.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:04 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:04.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:04 smithi143 ceph-mon[35860]: pgmap v48: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:04.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:04 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:04.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:04 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:04.842 DEBUG:teuthology.orchestra.run.smithi105:osd.1> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.1.service 2024-06-15T00:51:04.845 INFO:tasks.cephadm:Deploying osd.2 on smithi105 with /dev/vg_nvme/lv_2... 2024-06-15T00:51:04.845 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-15T00:51:05.395 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:05 smithi105 ceph-mon[28877]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-15T00:51:05.395 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:05 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1[54509]: 2024-06-15T00:51:05.087+0000 7f90284ff2c0 -1 osd.1 0 log_to_monitors true 2024-06-15T00:51:05.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:05 smithi143 ceph-mon[35860]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]: dispatch 2024-06-15T00:51:05.732 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:51:06.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-mon[28877]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:06.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-mon[28877]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-15T00:51:06.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-mon[28877]: osdmap e12: 2 total, 1 up, 2 in 2024-06-15T00:51:06.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:06.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-mon[28877]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-06-15T00:51:06.527 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1[54509]: 2024-06-15T00:51:06.336+0000 7f9023c71640 -1 osd.1 0 waiting for initial osdmap 2024-06-15T00:51:06.527 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:06 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1[54509]: 2024-06-15T00:51:06.341+0000 7f90202ab640 -1 osd.1 13 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:51:06.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:06 smithi143 ceph-mon[35860]: pgmap v49: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:06.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:06 smithi143 ceph-mon[35860]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["1"]}]': finished 2024-06-15T00:51:06.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:06 smithi143 ceph-mon[35860]: osdmap e12: 2 total, 1 up, 2 in 2024-06-15T00:51:06.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:06 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:06.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:06 smithi143 ceph-mon[35860]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd=[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-06-15T00:51:07.439 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:07 smithi105 ceph-mon[28877]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-06-15T00:51:07.439 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:07 smithi105 ceph-mon[28877]: osdmap e13: 2 total, 1 up, 2 in 2024-06-15T00:51:07.439 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:07 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:07.439 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:07 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:07.439 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:07 smithi105 ceph-mon[28877]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:07.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:07 smithi143 ceph-mon[35860]: from='osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114]' entity='osd.1' cmd='[{"prefix": "osd crush create-or-move", "id": 1, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-06-15T00:51:07.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:07 smithi143 ceph-mon[35860]: osdmap e13: 2 total, 1 up, 2 in 2024-06-15T00:51:07.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:07 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:07.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:07 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:07.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:07 smithi143 ceph-mon[35860]: pgmap v52: 0 pgs: ; 0 B data, 290 MiB used, 89 GiB / 89 GiB avail 2024-06-15T00:51:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-15T00:51:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-15T00:51:08.478 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10+0 records in 2024-06-15T00:51:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:10+0 records out 2024-06-15T00:51:08.478 INFO:teuthology.orchestra.run.smithi105.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.012507 s, 838 MB/s 2024-06-15T00:51:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:--> Zapping successful for: 2024-06-15T00:51:08.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:08 smithi143 ceph-mon[35860]: purged_snaps scrub starts 2024-06-15T00:51:08.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:08 smithi143 ceph-mon[35860]: purged_snaps scrub ok 2024-06-15T00:51:08.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:08 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:08.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:08 smithi143 ceph-mon[35860]: osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] boot 2024-06-15T00:51:08.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:08 smithi143 ceph-mon[35860]: osdmap e14: 2 total, 2 up, 2 in 2024-06-15T00:51:08.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:08 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:08.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:08 smithi105 ceph-mon[28877]: purged_snaps scrub starts 2024-06-15T00:51:08.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:08 smithi105 ceph-mon[28877]: purged_snaps scrub ok 2024-06-15T00:51:08.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:08 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:08.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:08 smithi105 ceph-mon[28877]: osd.1 [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] boot 2024-06-15T00:51:08.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:08 smithi105 ceph-mon[28877]: osdmap e14: 2 total, 2 up, 2 in 2024-06-15T00:51:08.749 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:08 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 1}]: dispatch 2024-06-15T00:51:09.132 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi105:vg_nvme/lv_2 2024-06-15T00:51:09.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:09 smithi105 ceph-mon[28877]: osdmap e15: 2 total, 2 up, 2 in 2024-06-15T00:51:09.365 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:09 smithi105 ceph-mon[28877]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:09.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:09 smithi143 ceph-mon[35860]: osdmap e15: 2 total, 2 up, 2 in 2024-06-15T00:51:09.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:09 smithi143 ceph-mon[35860]: pgmap v55: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:09.823 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:51:10.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:10.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:10.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:11.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:11 smithi143 ceph-mon[35860]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:11.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:11 smithi105 ceph-mon[28877]: pgmap v56: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:12.274 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 -- 172.21.15.105:0/3729965933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 msgr2=0x7fea680fb5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/3729965933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea680fb5c0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 crypto rx=0x7fea640048d0 tx=0x7fea64009da0 comp rx=0 tx=0).stop 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 -- 172.21.15.105:0/3729965933 shutdown_connections 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/3729965933 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea680fc480 0x7fea680fc8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/3729965933 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea680fb5c0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 -- 172.21.15.105:0/3729965933 >> 172.21.15.105:0/3729965933 conn(0x7fea680f6970 msgr2=0x7fea680f8da0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 -- 172.21.15.105:0/3729965933 shutdown_connections 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 -- 172.21.15.105:0/3729965933 wait complete. 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.246+0000 7fea6f6f9640 1 Processor -- start 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6f6f9640 1 -- start start 2024-06-15T00:51:12.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6f6f9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea6810bdb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6f6f9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea680fc480 0x7fea6810c2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6f6f9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6810c940 con 0x7fea680fb1b0 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6f6f9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea6810d080 con 0x7fea680fc480 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea6810bdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea6810bdb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43780/0 (socket says 172.21.15.105:43780) 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 -- 172.21.15.105:0/626475676 learned_addr learned my addr 172.21.15.105:0/626475676 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6cc94640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea680fc480 0x7fea6810c2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 -- 172.21.15.105:0/626475676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea680fc480 msgr2=0x7fea6810c2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea680fc480 0x7fea6810c2f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 -- 172.21.15.105:0/626475676 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea64005040 con 0x7fea680fb1b0 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.247+0000 7fea6d495640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea6810bdb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fea6400b020 tx=0x7fea64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.248+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea64038070 con 0x7fea680fb1b0 2024-06-15T00:51:12.276 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.248+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea6810d2b0 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.248+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea64032d30 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.248+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fea640097e0 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.248+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea6810d7f0 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.249+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea680fa610 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.249+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fea64030020 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.250+0000 7fea5dffb640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fea540653a0 0x7fea54067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.250+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 2410+0+0 (secure 0 0 0) 0x7fea640a2130 con 0x7fea680fb1b0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.250+0000 7fea6cc94640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fea540653a0 0x7fea54067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.250+0000 7fea6cc94640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fea540653a0 0x7fea54067850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea580041d0 tx=0x7fea58009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:51:12.277 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.254+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fea64070f70 con 0x7fea680fb1b0 2024-06-15T00:51:12.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:12.431+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fea6810dbc0 con 0x7fea540653a0 2024-06-15T00:51:12.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:12 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:51:12.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:12 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:51:12.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:12 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:12.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:12 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:51:12.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:12 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:51:12.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:12 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:13.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:13 smithi105 ceph-mon[28877]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:51:13.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:13 smithi105 ceph-mon[28877]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:13.791 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:13 smithi143 ceph-mon[35860]: from='client.14282 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi105:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:51:13.791 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:13 smithi143 ceph-mon[35860]: pgmap v57: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: Detected new or changed devices on smithi105 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: Adjusting osd_memory_target on smithi105 to 6432M 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:15.958 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:15.959 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:15.959 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:15.959 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:15 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:16.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: pgmap v58: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: Detected new or changed devices on smithi105 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: Adjusting osd_memory_target on smithi105 to 6432M 2024-06-15T00:51:16.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:16.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:16.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:16.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:16.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:15 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:17.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:16 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/727971662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6200bbe6-b8df-4a68-92b1-10f9544a308e"}]: dispatch 2024-06-15T00:51:17.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:16 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/727971662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6200bbe6-b8df-4a68-92b1-10f9544a308e"}]': finished 2024-06-15T00:51:17.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:16 smithi105 ceph-mon[28877]: osdmap e16: 3 total, 2 up, 3 in 2024-06-15T00:51:17.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:16 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:17.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:16 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/4236272960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:51:17.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:16 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/727971662' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "6200bbe6-b8df-4a68-92b1-10f9544a308e"}]: dispatch 2024-06-15T00:51:17.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:16 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/727971662' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "6200bbe6-b8df-4a68-92b1-10f9544a308e"}]': finished 2024-06-15T00:51:17.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:16 smithi143 ceph-mon[35860]: osdmap e16: 3 total, 2 up, 3 in 2024-06-15T00:51:17.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:16 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:17.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:16 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/4236272960' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:51:18.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:17 smithi105 ceph-mon[28877]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:18.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:17 smithi143 ceph-mon[35860]: pgmap v60: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:20.234 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:19 smithi105 ceph-mon[28877]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:20.234 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:19 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:20.234 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:19 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:20.234 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:19 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:20.235 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:19 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:20.235 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:19 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:19 smithi143 ceph-mon[35860]: pgmap v61: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:19 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:19 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:19 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:19 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:20.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:19 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:22.231 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:21 smithi105 ceph-mon[28877]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:22.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:21 smithi143 ceph-mon[35860]: pgmap v62: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:24.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:23 smithi105 ceph-mon[28877]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:23 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-15T00:51:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:23 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:23 smithi105 ceph-mon[28877]: Deploying daemon osd.2 on smithi105 2024-06-15T00:51:24.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:23 smithi143 ceph-mon[35860]: pgmap v63: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:24.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:23 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.2"}]: dispatch 2024-06-15T00:51:24.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:23 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:24.387 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:23 smithi143 ceph-mon[35860]: Deploying daemon osd.2 on smithi105 2024-06-15T00:51:25.182 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:25.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:24 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:26.266 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:25 smithi105 ceph-mon[28877]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:26.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:25 smithi143 ceph-mon[35860]: pgmap v64: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:28.238 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:27 smithi105 ceph-mon[28877]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:28.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:27 smithi143 ceph-mon[35860]: pgmap v65: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:29.225 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:29.225 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:29.225 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:28 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:29.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:29.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:29.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:28 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:30.194 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:29 smithi105 ceph-mon[28877]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:30.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:29 smithi143 ceph-mon[35860]: pgmap v66: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:32.124 INFO:teuthology.orchestra.run.smithi105.stdout:Created osd(s) 2 on host 'smithi105' 2024-06-15T00:51:32.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.123+0000 7fea5dffb640 1 -- 172.21.15.105:0/626475676 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fea6810dbc0 con 0x7fea540653a0 2024-06-15T00:51:32.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fea540653a0 msgr2=0x7fea54067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:32.125 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fea540653a0 0x7fea54067850 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 crypto rx=0x7fea580041d0 tx=0x7fea58009040 comp rx=0 tx=0).stop 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 msgr2=0x7fea6810bdb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea6810bdb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 crypto rx=0x7fea6400b020 tx=0x7fea64009da0 comp rx=0 tx=0).stop 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 shutdown_connections 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fea540653a0 0x7fea54067850 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fea680fc480 0x7fea6810c2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 --2- 172.21.15.105:0/626475676 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fea680fb1b0 0x7fea6810bdb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 >> 172.21.15.105:0/626475676 conn(0x7fea680f6970 msgr2=0x7fea680f8170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 shutdown_connections 2024-06-15T00:51:32.126 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:51:32.125+0000 7fea6f6f9640 1 -- 172.21.15.105:0/626475676 wait complete. 2024-06-15T00:51:32.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:31 smithi105 ceph-mon[28877]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:32.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:31 smithi143 ceph-mon[35860]: pgmap v67: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:33.063 DEBUG:teuthology.orchestra.run.smithi105:osd.2> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.2.service 2024-06-15T00:51:33.066 INFO:tasks.cephadm:Deploying osd.3 on smithi143 with /dev/vg_nvme/lv_4... 2024-06-15T00:51:33.066 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_4 2024-06-15T00:51:33.327 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:51:33.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:33 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:33.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:33 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:33.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:33 smithi143 ceph-mon[35860]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-15T00:51:33.450 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:33 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:33.450 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:33 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:33.450 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:33 smithi105 ceph-mon[28877]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]: dispatch 2024-06-15T00:51:34.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-mon[28877]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-mon[28877]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-15T00:51:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-mon[28877]: osdmap e17: 3 total, 2 up, 3 in 2024-06-15T00:51:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-mon[28877]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-06-15T00:51:34.277 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2[62195]: 2024-06-15T00:51:34.135+0000 7f4414a91640 -1 osd.2 0 waiting for initial osdmap 2024-06-15T00:51:34.277 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:34 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2[62195]: 2024-06-15T00:51:34.139+0000 7f440e8c6640 -1 osd.2 18 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:51:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:34 smithi143 ceph-mon[35860]: pgmap v68: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:34 smithi143 ceph-mon[35860]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["2"]}]': finished 2024-06-15T00:51:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:34 smithi143 ceph-mon[35860]: osdmap e17: 3 total, 2 up, 3 in 2024-06-15T00:51:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:34 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:34.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:34 smithi143 ceph-mon[35860]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd=[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]: dispatch 2024-06-15T00:51:34.966 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_4 2024-06-15T00:51:35.048 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_4 bs=1M count=10 conv=fsync 2024-06-15T00:51:35.061 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-06-15T00:51:35.061 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-06-15T00:51:35.062 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0114128 s, 919 MB/s 2024-06-15T00:51:35.062 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-06-15T00:51:35.352 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:35 smithi143 ceph-mon[35860]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-06-15T00:51:35.352 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:35 smithi143 ceph-mon[35860]: osdmap e18: 3 total, 2 up, 3 in 2024-06-15T00:51:35.352 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:35 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:35.352 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:35 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:35.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:35 smithi105 ceph-mon[28877]: from='osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438]' entity='osd.2' cmd='[{"prefix": "osd crush create-or-move", "id": 2, "weight":0.0873, "args": ["host=smithi105", "root=default"]}]': finished 2024-06-15T00:51:35.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:35 smithi105 ceph-mon[28877]: osdmap e18: 3 total, 2 up, 3 in 2024-06-15T00:51:35.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:35 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:35.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:35 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:35.720 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_4 2024-06-15T00:51:35.974 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: purged_snaps scrub starts 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: purged_snaps scrub ok 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] boot 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: osdmap e19: 3 total, 3 up, 3 in 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:36.373 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:36 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: purged_snaps scrub starts 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: purged_snaps scrub ok 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: pgmap v71: 0 pgs: ; 0 B data, 581 MiB used, 178 GiB / 179 GiB avail 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: osd.2 [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] boot 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: osdmap e19: 3 total, 3 up, 3 in 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 2}]: dispatch 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:36.418 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:36 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:37.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:37 smithi143 ceph-mon[35860]: osdmap e20: 3 total, 3 up, 3 in 2024-06-15T00:51:37.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:37 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-15T00:51:37.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/916450811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 msgr2=0x7fbdf00fc5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:37.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/916450811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf00fc5f0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 crypto rx=0x7fbdd80048d0 tx=0x7fbdd8009da0 comp rx=0 tx=0).stop 2024-06-15T00:51:37.420 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/916450811 shutdown_connections 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/916450811 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf00fc5f0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/916450811 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbdf00fb830 0x7fbdf00fbc40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/916450811 >> 172.21.15.143:0/916450811 conn(0x7fbdf00f6100 msgr2=0x7fbdf00f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/916450811 shutdown_connections 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/916450811 wait complete. 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 Processor -- start 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.417+0000 7fbdf4fe5640 1 -- start start 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdf4fe5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbdf00fb830 0x7fbdf010bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdf4fe5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf010c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdf4fe5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf010cfe0 con 0x7fbdf00fb830 2024-06-15T00:51:37.421 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdf4fe5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdf010d150 con 0x7fbdf00fc180 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf010c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf010c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:48572/0 (socket says 172.21.15.143:48572) 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 -- 172.21.15.143:0/117550114 learned_addr learned my addr 172.21.15.143:0/117550114 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdee59c640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbdf00fb830 0x7fbdf010bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 -- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbdf00fb830 msgr2=0x7fbdf010bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbdf00fb830 0x7fbdf010bd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 -- 172.21.15.143:0/117550114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8005040 con 0x7fbdf00fc180 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde7fff640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf010c2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8004930 tx=0x7fbdd8009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdd8038070 con 0x7fbdf00fc180 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdf010d3d0 con 0x7fbdf00fc180 2024-06-15T00:51:37.422 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdd8032d40 con 0x7fbdf00fc180 2024-06-15T00:51:37.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbdd803eb60 con 0x7fbdf00fc180 2024-06-15T00:51:37.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.418+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdf010d970 con 0x7fbdf00fc180 2024-06-15T00:51:37.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.420+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdf00f9da0 con 0x7fbdf00fc180 2024-06-15T00:51:37.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.420+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 18) v1 ==== 83691+0+0 (secure 0 0 0) 0x7fbdd8030020 con 0x7fbdf00fc180 2024-06-15T00:51:37.423 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.420+0000 7fbde6ffd640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbddc0653a0 0x7fbddc067850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:51:37.424 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.420+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 3217+0+0 (secure 0 0 0) 0x7fbdd80a2520 con 0x7fbdf00fc180 2024-06-15T00:51:37.424 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.420+0000 7fbdee59c640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbddc0653a0 0x7fbddc067850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:51:37.424 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.421+0000 7fbdee59c640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbddc0653a0 0x7fbddc067850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009af0 tx=0x7fbde000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:51:37.425 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.425+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbdd8071030 con 0x7fbdf00fc180 2024-06-15T00:51:37.596 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:37.595+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}) v1 -- 0x7fbdf0103dc0 con 0x7fbddc0653a0 2024-06-15T00:51:37.635 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:37 smithi105 ceph-mon[28877]: osdmap e20: 3 total, 3 up, 3 in 2024-06-15T00:51:37.635 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:37 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]: dispatch 2024-06-15T00:51:38.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:38.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-15T00:51:38.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: osdmap e21: 3 total, 3 up, 3 in 2024-06-15T00:51:38.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-15T00:51:38.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:51:38.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:51:38.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:51:38.599 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:38 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:38.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: pgmap v74: 0 pgs: ; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:38.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "osd pool create", "format": "json", "pool": ".mgr", "pg_num": 1, "pg_num_min": 1, "pg_num_max": 32}]': finished 2024-06-15T00:51:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: osdmap e21: 3 total, 3 up, 3 in 2024-06-15T00:51:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]: dispatch 2024-06-15T00:51:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: from='client.24127 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_4", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:51:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:51:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:51:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:38 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:38 smithi105 sudo[66413]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:51:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:38 smithi105 sudo[66413]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:38 smithi105 sudo[66413]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:39.146 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:38 smithi105 sudo[66413]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:39.361 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:39.360+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fbdd8071400 con 0x7fbdf00fc180 2024-06-15T00:51:39.413 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66448]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:51:39.413 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66448]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:39.414 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66448]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:39.414 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66448]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:39.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-15T00:51:39.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: osdmap e22: 3 total, 3 up, 3 in 2024-06-15T00:51:39.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:39.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: Detected new or changed devices on smithi105 2024-06-15T00:51:39.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.414 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: Adjusting osd_memory_target on smithi105 to 4288M 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:39.415 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.600 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd='[{"prefix": "osd pool application enable", "format": "json", "pool": ".mgr", "app": "mgr", "yes_i_really_mean_it": true}]': finished 2024-06-15T00:51:39.600 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: osdmap e22: 3 total, 3 up, 3 in 2024-06-15T00:51:39.600 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: pgmap v77: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:39.600 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: Detected new or changed devices on smithi105 2024-06-15T00:51:39.600 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.0", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.1", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.2", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: Adjusting osd_memory_target on smithi105 to 4288M 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:39.601 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:39.673 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66451]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:51:39.673 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66451]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:39.673 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66451]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:39.673 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66451]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:39.935 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66489]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:51:39.935 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66489]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:39.935 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66489]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:39.935 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66489]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:39.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66486]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:51:39.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66486]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:39.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66486]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:39.936 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:51:39 smithi105 sudo[66486]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:40.276 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66492]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:51:40.277 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66492]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:40.277 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66492]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:40.277 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66492]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:40.685 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66495]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:51:40.686 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66495]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:40.686 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66495]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:40.686 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66495]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66498]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:51:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66498]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66498]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 sudo[66498]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:41.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: osdmap e23: 3 total, 3 up, 3 in 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: mgrmap e19: smithi105.oorfym(active, since 2m), standbys: smithi143.hhtfap 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1185771550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2ee8545-5b79-454a-97bc-a3975c9d1528"}]: dispatch 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2ee8545-5b79-454a-97bc-a3975c9d1528"}]: dispatch 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2ee8545-5b79-454a-97bc-a3975c9d1528"}]': finished 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: osdmap e24: 4 total, 3 up, 4 in 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:41.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:40 smithi105 ceph-mon[28877]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:51:41.228 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:40 smithi143 sudo[39425]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/smartctl -x --json=o /dev/nvme0n1 2024-06-15T00:51:41.228 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:40 smithi143 sudo[39425]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:41.228 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:40 smithi143 sudo[39425]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:41.228 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:40 smithi143 sudo[39425]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 sudo[39429]: ceph : PWD=/ ; USER=root ; COMMAND=/usr/sbin/nvme intel smart-log-add --json /dev/nvme0n1 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 sudo[39429]: pam_systemd(sudo:session): Failed to connect to system bus: No such file or directory 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 sudo[39429]: pam_unix(sudo:session): session opened for user root(uid=0) by (uid=167) 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 sudo[39429]: pam_unix(sudo:session): session closed for user root 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: osdmap e23: 3 total, 3 up, 3 in 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: mgrmap e19: smithi105.oorfym(active, since 2m), standbys: smithi143.hhtfap 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1185771550' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2ee8545-5b79-454a-97bc-a3975c9d1528"}]: dispatch 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "b2ee8545-5b79-454a-97bc-a3975c9d1528"}]: dispatch 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "b2ee8545-5b79-454a-97bc-a3975c9d1528"}]': finished 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: osdmap e24: 4 total, 3 up, 4 in 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:41.637 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:51:42.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/4143821676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:51:42.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:41 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:42.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/4143821676' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: pgmap v80: 1 pgs: 1 creating+peering; 0 B data, 872 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='admin socket' entity='admin socket' cmd='smart' args=[json]: dispatch 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi105"}]: dispatch 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "mon metadata", "id": "smithi143"}]: dispatch 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='admin socket' entity='admin socket' cmd=smart args=[json]: finished 2024-06-15T00:51:42.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:41 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:43.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:43 smithi143 ceph-mon[35860]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:44.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:43 smithi105 ceph-mon[28877]: pgmap v81: 1 pgs: 1 creating+peering; 0 B data, 873 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:45.798 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:45 smithi143 ceph-mon[35860]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:46.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:45 smithi105 ceph-mon[28877]: pgmap v82: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:47 smithi105 ceph-mon[28877]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-15T00:51:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:48.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:47 smithi105 ceph-mon[28877]: Deploying daemon osd.3 on smithi143 2024-06-15T00:51:48.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:47 smithi143 ceph-mon[35860]: pgmap v83: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:48.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.3"}]: dispatch 2024-06-15T00:51:48.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:48.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:47 smithi143 ceph-mon[35860]: Deploying daemon osd.3 on smithi143 2024-06-15T00:51:49.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:49 smithi143 ceph-mon[35860]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:50.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:49 smithi105 ceph-mon[28877]: pgmap v84: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:52.025 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:51 smithi143 ceph-mon[35860]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:51 smithi105 ceph-mon[28877]: pgmap v85: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:52.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:54.015 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:53 smithi143 ceph-mon[35860]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:54.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:53 smithi105 ceph-mon[28877]: pgmap v86: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:55.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:55.070 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:51:55.708 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 3 on host 'smithi143' 2024-06-15T00:51:55.709 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.708+0000 7fbde6ffd640 1 -- 172.21.15.143:0/117550114 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbdf0103dc0 con 0x7fbddc0653a0 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbddc0653a0 msgr2=0x7fbddc067850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbddc0653a0 0x7fbddc067850 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 crypto rx=0x7fbde0009af0 tx=0x7fbde000b040 comp rx=0 tx=0).stop 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 msgr2=0x7fbdf010c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf010c2b0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0x7fbdd8004930 tx=0x7fbdd8009da0 comp rx=0 tx=0).stop 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 shutdown_connections 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbddc0653a0 0x7fbddc067850 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbdf00fc180 0x7fbdf010c2b0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 --2- 172.21.15.143:0/117550114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbdf00fb830 0x7fbdf010bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 >> 172.21.15.143:0/117550114 conn(0x7fbdf00f6100 msgr2=0x7fbdf00f78a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 shutdown_connections 2024-06-15T00:51:55.711 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:51:55.711+0000 7fbdf4fe5640 1 -- 172.21.15.143:0/117550114 wait complete. 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.265 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='osd.3 [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:51:56.266 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:51:56.266 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:56.266 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.266 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: pgmap v87: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='osd.3 [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997]' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]: dispatch 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:51:56.294 DEBUG:teuthology.orchestra.run.smithi143:osd.3> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.3.service 2024-06-15T00:51:56.296 INFO:tasks.cephadm:Deploying osd.4 on smithi143 with /dev/vg_nvme/lv_3... 2024-06-15T00:51:56.296 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_3 2024-06-15T00:51:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:56 smithi105 ceph-mon[28877]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-15T00:51:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:56 smithi105 ceph-mon[28877]: osdmap e25: 4 total, 3 up, 4 in 2024-06-15T00:51:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:56 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:56 smithi105 ceph-mon[28877]: from='osd.3 [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:51:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:56 smithi105 ceph-mon[28877]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:51:57.289 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-mon[35860]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["3"]}]': finished 2024-06-15T00:51:57.289 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-mon[35860]: osdmap e25: 4 total, 3 up, 4 in 2024-06-15T00:51:57.289 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:57.289 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-mon[35860]: from='osd.3 [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997]' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:51:57.289 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-mon[35860]: from='osd.3 ' entity='osd.3' cmd=[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:51:57.289 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3[41369]: 2024-06-15T00:51:56.986+0000 7fc457926640 -1 osd.3 0 waiting for initial osdmap 2024-06-15T00:51:57.289 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:51:56 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3[41369]: 2024-06-15T00:51:56.991+0000 7fc45175b640 -1 osd.3 26 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:51:57.351 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:51:58.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:57 smithi105 ceph-mon[28877]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:58.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:57 smithi105 ceph-mon[28877]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-06-15T00:51:58.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:57 smithi105 ceph-mon[28877]: osdmap e26: 4 total, 3 up, 4 in 2024-06-15T00:51:58.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:57 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:58.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:57 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:58.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:57 smithi143 ceph-mon[35860]: pgmap v89: 1 pgs: 1 active+clean; 577 KiB data, 874 MiB used, 267 GiB / 268 GiB avail 2024-06-15T00:51:58.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:57 smithi143 ceph-mon[35860]: from='osd.3 ' entity='osd.3' cmd='[{"prefix": "osd crush create-or-move", "id": 3, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-06-15T00:51:58.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:57 smithi143 ceph-mon[35860]: osdmap e26: 4 total, 3 up, 4 in 2024-06-15T00:51:58.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:57 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:58.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:57 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:59.019 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:58 smithi143 ceph-mon[35860]: purged_snaps scrub starts 2024-06-15T00:51:59.020 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:58 smithi143 ceph-mon[35860]: purged_snaps scrub ok 2024-06-15T00:51:59.020 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:58 smithi143 ceph-mon[35860]: osd.3 [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] boot 2024-06-15T00:51:59.020 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:58 smithi143 ceph-mon[35860]: osdmap e27: 4 total, 4 up, 4 in 2024-06-15T00:51:59.020 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:58 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:59.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:58 smithi105 ceph-mon[28877]: purged_snaps scrub starts 2024-06-15T00:51:59.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:58 smithi105 ceph-mon[28877]: purged_snaps scrub ok 2024-06-15T00:51:59.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:58 smithi105 ceph-mon[28877]: osd.3 [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] boot 2024-06-15T00:51:59.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:58 smithi105 ceph-mon[28877]: osdmap e27: 4 total, 4 up, 4 in 2024-06-15T00:51:59.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:58 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 3}]: dispatch 2024-06-15T00:51:59.341 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_3 2024-06-15T00:51:59.432 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_3 bs=1M count=10 conv=fsync 2024-06-15T00:51:59.447 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-06-15T00:51:59.447 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-06-15T00:51:59.447 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0139163 s, 753 MB/s 2024-06-15T00:51:59.447 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-06-15T00:51:59.966 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_3 2024-06-15T00:52:00.109 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:59 smithi143 ceph-mon[35860]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:00.110 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:59 smithi143 ceph-mon[35860]: osdmap e28: 4 total, 4 up, 4 in 2024-06-15T00:52:00.110 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:51:59 smithi143 ceph-mon[35860]: osdmap e29: 4 total, 4 up, 4 in 2024-06-15T00:52:00.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:59 smithi105 ceph-mon[28877]: pgmap v92: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:00.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:59 smithi105 ceph-mon[28877]: osdmap e28: 4 total, 4 up, 4 in 2024-06-15T00:52:00.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:51:59 smithi105 ceph-mon[28877]: osdmap e29: 4 total, 4 up, 4 in 2024-06-15T00:52:00.454 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:52:01.721 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: Detected new or changed devices on smithi143 2024-06-15T00:52:01.721 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.721 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.721 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:01.721 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: Adjusting osd_memory_target on smithi143 to 15936M 2024-06-15T00:52:01.722 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.722 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:01.722 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:01.722 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.722 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: osdmap e30: 4 total, 4 up, 4 in 2024-06-15T00:52:01.722 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:01 smithi143 ceph-mon[35860]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:01.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: Detected new or changed devices on smithi143 2024-06-15T00:52:01.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:01.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: Adjusting osd_memory_target on smithi143 to 15936M 2024-06-15T00:52:01.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:01.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:01.778 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:01.779 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: osdmap e30: 4 total, 4 up, 4 in 2024-06-15T00:52:01.779 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:01 smithi105 ceph-mon[28877]: pgmap v96: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:01.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 -- 172.21.15.143:0/3767556313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 msgr2=0x7f0f5c0fd9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:01.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3767556313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c0fd9d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c0048d0 tx=0x7f0f4c009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:01.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 -- 172.21.15.143:0/3767556313 shutdown_connections 2024-06-15T00:52:01.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3767556313 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c0fd9d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:01.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3767556313 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f5c0fc290 0x7f0f5c0fc6a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:01.922 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 -- 172.21.15.143:0/3767556313 >> 172.21.15.143:0/3767556313 conn(0x7f0f5c0f7a70 msgr2=0x7f0f5c0f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 -- 172.21.15.143:0/3767556313 shutdown_connections 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.919+0000 7f0f6329a640 1 -- 172.21.15.143:0/3767556313 wait complete. 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f6329a640 1 Processor -- start 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f6329a640 1 -- start start 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f6329a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f5c0fc290 0x7f0f5c10ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f6329a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c10d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f6329a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c10da10 con 0x7f0f5c0fc290 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f6329a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f5c10e100 con 0x7f0f5c0fd560 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f60835640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c10d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f60835640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c10d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:50788/0 (socket says 172.21.15.143:50788) 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f60835640 1 -- 172.21.15.143:0/3184845383 learned_addr learned my addr 172.21.15.143:0/3184845383 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.920+0000 7f0f61036640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f5c0fc290 0x7f0f5c10ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:01.923 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f60835640 1 -- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f5c0fc290 msgr2=0x7f0f5c10ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f60835640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f5c0fc290 0x7f0f5c10ce80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f60835640 1 -- 172.21.15.143:0/3184845383 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f4c005040 con 0x7f0f5c0fd560 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f60835640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c10d3c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c02ccc0 tx=0x7f0f4c009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f4c038070 con 0x7f0f5c0fd560 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f4c02cea0 con 0x7f0f5c0fd560 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f5c10e330 con 0x7f0f5c0fd560 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0f4c032df0 con 0x7f0f5c0fd560 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.921+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f5c10e870 con 0x7f0f5c0fd560 2024-06-15T00:52:01.924 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.922+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f5c0fb6f0 con 0x7f0f5c0fd560 2024-06-15T00:52:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.923+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0f4c030020 con 0x7f0f5c0fd560 2024-06-15T00:52:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.924+0000 7f0f51ffb640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0f48065430 0x7f0f480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.924+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 3770+0+0 (secure 0 0 0) 0x7f0f4c0a27e0 con 0x7f0f5c0fd560 2024-06-15T00:52:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.924+0000 7f0f61036640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0f48065430 0x7f0f480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:01.925 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.924+0000 7f0f61036640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0f48065430 0x7f0f480678e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c10cc10 tx=0x7f0f5800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:01.927 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:01.927+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0f4c0711f0 con 0x7f0f5c0fd560 2024-06-15T00:52:02.099 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:02.099+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f5c10ecc0 con 0x7f0f48065430 2024-06-15T00:52:03.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:02 smithi105 ceph-mon[28877]: osdmap e31: 4 total, 4 up, 4 in 2024-06-15T00:52:03.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:02 smithi105 ceph-mon[28877]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:52:03.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:02 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:52:03.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:02 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:52:03.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:02 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:03.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:02 smithi143 ceph-mon[35860]: osdmap e31: 4 total, 4 up, 4 in 2024-06-15T00:52:03.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:02 smithi143 ceph-mon[35860]: from='client.24149 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_3", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:52:03.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:02 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:52:03.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:02 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:52:03.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:02 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:03.806 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:03 smithi143 ceph-mon[35860]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:04.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:03 smithi105 ceph-mon[28877]: pgmap v98: 1 pgs: 1 remapped+peering; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:04.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:04 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/4042649351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bda317b1-8168-4cb1-9593-c12cc95a00dd"}]: dispatch 2024-06-15T00:52:04.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:04 smithi105 ceph-mon[28877]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bda317b1-8168-4cb1-9593-c12cc95a00dd"}]: dispatch 2024-06-15T00:52:04.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:04 smithi105 ceph-mon[28877]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bda317b1-8168-4cb1-9593-c12cc95a00dd"}]': finished 2024-06-15T00:52:04.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:04 smithi105 ceph-mon[28877]: osdmap e32: 5 total, 4 up, 5 in 2024-06-15T00:52:04.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:04 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:05.101 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:04 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/4042649351' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bda317b1-8168-4cb1-9593-c12cc95a00dd"}]: dispatch 2024-06-15T00:52:05.101 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:04 smithi143 ceph-mon[35860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "bda317b1-8168-4cb1-9593-c12cc95a00dd"}]: dispatch 2024-06-15T00:52:05.101 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:04 smithi143 ceph-mon[35860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "bda317b1-8168-4cb1-9593-c12cc95a00dd"}]': finished 2024-06-15T00:52:05.101 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:04 smithi143 ceph-mon[35860]: osdmap e32: 5 total, 4 up, 5 in 2024-06-15T00:52:05.101 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:04 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:06.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:05 smithi105 ceph-mon[28877]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-06-15T00:52:06.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:05 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1915815315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:52:06.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:05 smithi143 ceph-mon[35860]: pgmap v100: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 122 KiB/s, 0 objects/s recovering 2024-06-15T00:52:06.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:05 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1915815315' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:52:08.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:07 smithi105 ceph-mon[28877]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-15T00:52:08.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:07 smithi143 ceph-mon[35860]: pgmap v101: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 96 KiB/s, 0 objects/s recovering 2024-06-15T00:52:09.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:09 smithi143 ceph-mon[35860]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-15T00:52:09.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:09 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:10.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:09 smithi105 ceph-mon[28877]: pgmap v102: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 72 KiB/s, 0 objects/s recovering 2024-06-15T00:52:10.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:09 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:11.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:11 smithi143 ceph-mon[35860]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-15T00:52:11.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:11 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-15T00:52:11.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:11 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:12.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:11 smithi105 ceph-mon[28877]: pgmap v103: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 64 KiB/s, 0 objects/s recovering 2024-06-15T00:52:12.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:11 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.4"}]: dispatch 2024-06-15T00:52:12.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:11 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:13.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:12 smithi105 ceph-mon[28877]: Deploying daemon osd.4 on smithi143 2024-06-15T00:52:13.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:12 smithi143 ceph-mon[35860]: Deploying daemon osd.4 on smithi143 2024-06-15T00:52:14.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:13 smithi105 ceph-mon[28877]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-15T00:52:14.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:13 smithi143 ceph-mon[35860]: pgmap v104: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail; 58 KiB/s, 0 objects/s recovering 2024-06-15T00:52:16.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:15 smithi105 ceph-mon[28877]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:16.057 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:15 smithi143 ceph-mon[35860]: pgmap v105: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:17.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:16 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:17.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:16 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:17.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:16 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:17.051 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:16 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:17.051 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:16 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:17.051 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:16 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:18.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:17 smithi105 ceph-mon[28877]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:18.076 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:17 smithi143 ceph-mon[35860]: pgmap v106: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:20.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:19 smithi105 ceph-mon[28877]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:20.099 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:19 smithi143 ceph-mon[35860]: pgmap v107: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:20.407 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 4 on host 'smithi143' 2024-06-15T00:52:20.407 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.406+0000 7f0f51ffb640 1 -- 172.21.15.143:0/3184845383 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7f0f5c10ecc0 con 0x7f0f48065430 2024-06-15T00:52:20.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0f48065430 msgr2=0x7f0f480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:20.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0f48065430 0x7f0f480678e0 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 crypto rx=0x7f0f5c10cc10 tx=0x7f0f5800b040 comp rx=0 tx=0).stop 2024-06-15T00:52:20.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 msgr2=0x7f0f5c10d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:20.409 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c10d3c0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 crypto rx=0x7f0f4c02ccc0 tx=0x7f0f4c009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 shutdown_connections 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0f48065430 0x7f0f480678e0 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0f5c0fd560 0x7f0f5c10d3c0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 --2- 172.21.15.143:0/3184845383 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0f5c0fc290 0x7f0f5c10ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 >> 172.21.15.143:0/3184845383 conn(0x7f0f5c0f7a70 msgr2=0x7f0f5c0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 shutdown_connections 2024-06-15T00:52:20.410 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:20.408+0000 7f0f6329a640 1 -- 172.21.15.143:0/3184845383 wait complete. 2024-06-15T00:52:21.019 DEBUG:teuthology.orchestra.run.smithi143:osd.4> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.4.service 2024-06-15T00:52:21.022 INFO:tasks.cephadm:Deploying osd.5 on smithi143 with /dev/vg_nvme/lv_2... 2024-06-15T00:52:21.022 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph-volume lvm zap /dev/vg_nvme/lv_2 2024-06-15T00:52:21.329 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:52:21.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: from='osd.4 [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:52:21.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:52:21.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:21.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:21 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: from='osd.4 [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185]' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:52:21.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]: dispatch 2024-06-15T00:52:21.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: pgmap v108: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:21.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:21.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:21 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:22.647 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-mon[35860]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-15T00:52:22.647 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-mon[35860]: osdmap e33: 5 total, 4 up, 5 in 2024-06-15T00:52:22.647 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:22.647 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-mon[35860]: from='osd.4 [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:22.647 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-mon[35860]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:22.647 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:52:22.425+0000 7ffa4bb45640 -1 osd.4 0 waiting for initial osdmap 2024-06-15T00:52:22.647 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:52:22 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:52:22.434+0000 7ffa44155640 -1 osd.4 34 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:52:22.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:22 smithi105 ceph-mon[28877]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["4"]}]': finished 2024-06-15T00:52:22.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:22 smithi105 ceph-mon[28877]: osdmap e33: 5 total, 4 up, 5 in 2024-06-15T00:52:22.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:22 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:22.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:22 smithi105 ceph-mon[28877]: from='osd.4 [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185]' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:22.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:22 smithi105 ceph-mon[28877]: from='osd.4 ' entity='osd.4' cmd=[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:23.634 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping: /dev/vg_nvme/lv_2 2024-06-15T00:52:23.634 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:23 smithi143 ceph-mon[35860]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-06-15T00:52:23.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:23 smithi143 ceph-mon[35860]: osdmap e34: 5 total, 4 up, 5 in 2024-06-15T00:52:23.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:23 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:23.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:23 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:23.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:23 smithi143 ceph-mon[35860]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:23.719 INFO:teuthology.orchestra.run.smithi143.stderr:Running command: /usr/bin/dd if=/dev/zero of=/dev/vg_nvme/lv_2 bs=1M count=10 conv=fsync 2024-06-15T00:52:23.734 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10+0 records in 2024-06-15T00:52:23.735 INFO:teuthology.orchestra.run.smithi143.stderr:10+0 records out 2024-06-15T00:52:23.735 INFO:teuthology.orchestra.run.smithi143.stderr: stderr: 10485760 bytes (10 MB, 10 MiB) copied, 0.0143814 s, 729 MB/s 2024-06-15T00:52:23.735 INFO:teuthology.orchestra.run.smithi143.stderr:--> Zapping successful for: 2024-06-15T00:52:23.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:23 smithi105 ceph-mon[28877]: from='osd.4 ' entity='osd.4' cmd='[{"prefix": "osd crush create-or-move", "id": 4, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-06-15T00:52:23.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:23 smithi105 ceph-mon[28877]: osdmap e34: 5 total, 4 up, 5 in 2024-06-15T00:52:23.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:23 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:23.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:23 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:23.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:23 smithi105 ceph-mon[28877]: pgmap v111: 1 pgs: 1 active+clean; 577 KiB data, 1.1 GiB used, 356 GiB / 358 GiB avail 2024-06-15T00:52:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:24 smithi105 ceph-mon[28877]: purged_snaps scrub starts 2024-06-15T00:52:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:24 smithi105 ceph-mon[28877]: purged_snaps scrub ok 2024-06-15T00:52:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:24 smithi105 ceph-mon[28877]: osd.4 [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] boot 2024-06-15T00:52:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:24 smithi105 ceph-mon[28877]: osdmap e35: 5 total, 5 up, 5 in 2024-06-15T00:52:24.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:24.808 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:24 smithi143 ceph-mon[35860]: purged_snaps scrub starts 2024-06-15T00:52:24.809 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:24 smithi143 ceph-mon[35860]: purged_snaps scrub ok 2024-06-15T00:52:24.809 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:24 smithi143 ceph-mon[35860]: osd.4 [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] boot 2024-06-15T00:52:24.809 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:24 smithi143 ceph-mon[35860]: osdmap e35: 5 total, 5 up, 5 in 2024-06-15T00:52:24.809 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:24 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 4}]: dispatch 2024-06-15T00:52:24.833 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph orch daemon add osd smithi143:vg_nvme/lv_2 2024-06-15T00:52:25.562 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:25 smithi143 ceph-mon[35860]: osdmap e36: 5 total, 5 up, 5 in 2024-06-15T00:52:25.563 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:25.563 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:25 smithi143 ceph-mon[35860]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:25.608 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:52:25.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:25 smithi105 ceph-mon[28877]: osdmap e36: 5 total, 5 up, 5 in 2024-06-15T00:52:25.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:25.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:25 smithi105 ceph-mon[28877]: pgmap v114: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: Detected new or changed devices on smithi143 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: Adjusting osd_memory_target on smithi143 to 7968M 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:26.881 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:26.882 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:26.882 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:26.882 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:26.882 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:26 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: Detected new or changed devices on smithi143 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: Adjusting osd_memory_target on smithi143 to 7968M 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:27.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:27.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:27.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:27.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:27.028 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:26 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:27.166 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 -- 172.21.15.143:0/771658258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 msgr2=0x7fbe1c0fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 --2- 172.21.15.143:0/771658258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c0fd9f0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 crypto rx=0x7fbe180048d0 tx=0x7fbe18009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 -- 172.21.15.143:0/771658258 shutdown_connections 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 --2- 172.21.15.143:0/771658258 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c0fd9f0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 --2- 172.21.15.143:0/771658258 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 0x7fbe1c0fc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 -- 172.21.15.143:0/771658258 >> 172.21.15.143:0/771658258 conn(0x7fbe1c0f7a90 msgr2=0x7fbe1c0f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 -- 172.21.15.143:0/771658258 shutdown_connections 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 -- 172.21.15.143:0/771658258 wait complete. 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.163+0000 7fbe236d2640 1 Processor -- start 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe236d2640 1 -- start start 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe236d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 0x7fbe1c10ced0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe236d2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c10d410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe236d2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c10e140 con 0x7fbe1c0fc2b0 2024-06-15T00:52:27.167 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe236d2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe1c10e2b0 con 0x7fbe1c0fd580 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c10d410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c10d410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:57774/0 (socket says 172.21.15.143:57774) 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 -- 172.21.15.143:0/4042684069 learned_addr learned my addr 172.21.15.143:0/4042684069 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe2146e640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 0x7fbe1c10ced0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 -- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 msgr2=0x7fbe1c10ced0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 0x7fbe1c10ced0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 -- 172.21.15.143:0/4042684069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe18005040 con 0x7fbe1c0fd580 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe2146e640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 0x7fbe1c10ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe20c6d640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c10d410 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009af0 tx=0x7fbe18009da0 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe18038070 con 0x7fbe1c0fd580 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe180096f0 con 0x7fbe1c0fd580 2024-06-15T00:52:27.168 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe1c10e530 con 0x7fbe1c0fd580 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.164+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbe18032e60 con 0x7fbe1c0fd580 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.165+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe1c10e9f0 con 0x7fbe1c0fd580 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.166+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe1c0fb710 con 0x7fbe1c0fd580 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.166+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fbe18030020 con 0x7fbe1c0fd580 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.167+0000 7fbe0dffb640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbe080653d0 0x7fbe08067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.167+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 4202+0+0 (secure 0 0 0) 0x7fbe180a2c70 con 0x7fbe1c0fd580 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.167+0000 7fbe2146e640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbe080653d0 0x7fbe08067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:27.169 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.167+0000 7fbe2146e640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbe080653d0 0x7fbe08067880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbe10006fd0 tx=0x7fbe10008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:27.170 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.171+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbe18071390 con 0x7fbe1c0fd580 2024-06-15T00:52:27.340 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:27.340+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}) v1 -- 0x7fbe1c10ec60 con 0x7fbe080653d0 2024-06-15T00:52:27.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:27 smithi143 ceph-mon[35860]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:27.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:27 smithi143 ceph-mon[35860]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:52:27.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:52:27.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:52:27.865 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:27 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:28.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:27 smithi105 ceph-mon[28877]: pgmap v115: 1 pgs: 1 peering; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:28.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:27 smithi105 ceph-mon[28877]: from='client.24167 -' entity='client.admin' cmd=[{"prefix": "orch daemon add osd", "svc_arg": "smithi143:vg_nvme/lv_2", "target": ["mon-mgr", ""]}]: dispatch 2024-06-15T00:52:28.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd tree", "states": ["destroyed"], "format": "json"}]: dispatch 2024-06-15T00:52:28.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.bootstrap-osd"}]: dispatch 2024-06-15T00:52:28.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:27 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:30.018 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:29 smithi143 ceph-mon[35860]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:30.018 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:29 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/2733913304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75426eb7-35b5-44f5-a6ce-ccb083273c69"}]: dispatch 2024-06-15T00:52:30.018 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:29 smithi143 ceph-mon[35860]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75426eb7-35b5-44f5-a6ce-ccb083273c69"}]: dispatch 2024-06-15T00:52:30.018 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:29 smithi143 ceph-mon[35860]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75426eb7-35b5-44f5-a6ce-ccb083273c69"}]': finished 2024-06-15T00:52:30.018 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:29 smithi143 ceph-mon[35860]: osdmap e37: 6 total, 5 up, 6 in 2024-06-15T00:52:30.018 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:29 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:30.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:29 smithi105 ceph-mon[28877]: pgmap v116: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:30.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:29 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/2733913304' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75426eb7-35b5-44f5-a6ce-ccb083273c69"}]: dispatch 2024-06-15T00:52:30.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:29 smithi105 ceph-mon[28877]: from='client.? ' entity='client.bootstrap-osd' cmd=[{"prefix": "osd new", "uuid": "75426eb7-35b5-44f5-a6ce-ccb083273c69"}]: dispatch 2024-06-15T00:52:30.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:29 smithi105 ceph-mon[28877]: from='client.? ' entity='client.bootstrap-osd' cmd='[{"prefix": "osd new", "uuid": "75426eb7-35b5-44f5-a6ce-ccb083273c69"}]': finished 2024-06-15T00:52:30.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:29 smithi105 ceph-mon[28877]: osdmap e37: 6 total, 5 up, 6 in 2024-06-15T00:52:30.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:29 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:31.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:30 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1644059034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:52:31.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:30 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1644059034' entity='client.bootstrap-osd' cmd=[{"prefix": "mon getmap"}]: dispatch 2024-06-15T00:52:32.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:31 smithi105 ceph-mon[28877]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:32.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:31 smithi143 ceph-mon[35860]: pgmap v118: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:34.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:33 smithi105 ceph-mon[28877]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:34.030 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:33 smithi143 ceph-mon[35860]: pgmap v119: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:35.857 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:35 smithi143 ceph-mon[35860]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:36.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:35 smithi105 ceph-mon[28877]: pgmap v120: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:37.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:37 smithi143 ceph-mon[35860]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:37.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:37 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-15T00:52:37.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:37 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:37.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:37 smithi143 ceph-mon[35860]: Deploying daemon osd.5 on smithi143 2024-06-15T00:52:38.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:37 smithi105 ceph-mon[28877]: pgmap v121: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:38.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:37 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "osd.5"}]: dispatch 2024-06-15T00:52:38.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:37 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:38.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:37 smithi105 ceph-mon[28877]: Deploying daemon osd.5 on smithi143 2024-06-15T00:52:40.101 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:39 smithi143 ceph-mon[35860]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:40.102 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:40.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:39 smithi105 ceph-mon[28877]: pgmap v122: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:42.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:41 smithi143 ceph-mon[35860]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:42.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:41 smithi105 ceph-mon[28877]: pgmap v123: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:42.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:42 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:42.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:42 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:42.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:42 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:43.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:42 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:43.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:42 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:43.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:42 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:44.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:43 smithi143 ceph-mon[35860]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:44.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:43 smithi105 ceph-mon[28877]: pgmap v124: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:45.397 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.397+0000 7fbe0dffb640 1 -- 172.21.15.143:0/4042684069 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+37 (secure 0 0 0) 0x7fbe1c10ec60 con 0x7fbe080653d0 2024-06-15T00:52:45.397 INFO:teuthology.orchestra.run.smithi143.stdout:Created osd(s) 5 on host 'smithi143' 2024-06-15T00:52:45.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbe080653d0 msgr2=0x7fbe08067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:45.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbe080653d0 0x7fbe08067880 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 crypto rx=0x7fbe10006fd0 tx=0x7fbe10008040 comp rx=0 tx=0).stop 2024-06-15T00:52:45.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 msgr2=0x7fbe1c10d410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:45.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c10d410 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 crypto rx=0x7fbe18009af0 tx=0x7fbe18009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:45.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 shutdown_connections 2024-06-15T00:52:45.399 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbe080653d0 0x7fbe08067880 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:45.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbe1c0fd580 0x7fbe1c10d410 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:45.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 --2- 172.21.15.143:0/4042684069 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbe1c0fc2b0 0x7fbe1c10ced0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:45.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 >> 172.21.15.143:0/4042684069 conn(0x7fbe1c0f7a90 msgr2=0x7fbe1c0f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:45.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 shutdown_connections 2024-06-15T00:52:45.400 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:52:45.399+0000 7fbe236d2640 1 -- 172.21.15.143:0/4042684069 wait complete. 2024-06-15T00:52:46.023 DEBUG:teuthology.orchestra.run.smithi143:osd.5> sudo journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.5.service 2024-06-15T00:52:46.026 INFO:tasks.cephadm:Waiting for 6 OSDs to come up... 2024-06-15T00:52:46.026 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-15T00:52:46.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:45 smithi105 ceph-mon[28877]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:45 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:45 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:45 smithi105 ceph-mon[28877]: from='osd.5 [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:52:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:45 smithi105 ceph-mon[28877]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:52:46.290 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:52:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:45 smithi143 ceph-mon[35860]: pgmap v125: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:45 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:45 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:46.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:45 smithi143 ceph-mon[35860]: from='osd.5 [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362]' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:52:46.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:45 smithi143 ceph-mon[35860]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]: dispatch 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: osdmap e38: 6 total, 5 up, 6 in 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: from='osd.5 [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:47.612 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-mon[35860]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:47.613 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:52:47.404+0000 7fcf34af6640 -1 osd.5 0 waiting for initial osdmap 2024-06-15T00:52:47.613 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:52:47 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:52:47.412+0000 7fcf3012e640 -1 osd.5 39 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2024-06-15T00:52:47.750 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush set-device-class", "class": "ssd", "ids": ["5"]}]': finished 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: osdmap e38: 6 total, 5 up, 6 in 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: from='osd.5 [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362]' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: from='osd.5 ' entity='osd.5' cmd=[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]: dispatch 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:47.751 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:47 smithi105 ceph-mon[28877]: pgmap v127: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:47.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.940+0000 7fed915cc640 1 -- 172.21.15.105:0/3440872301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fd580 msgr2=0x7fed8c0fd9f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:47.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.940+0000 7fed915cc640 1 --2- 172.21.15.105:0/3440872301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fd580 0x7fed8c0fd9f0 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 crypto rx=0x7fed740048d0 tx=0x7fed74009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:47.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- 172.21.15.105:0/3440872301 shutdown_connections 2024-06-15T00:52:47.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 --2- 172.21.15.105:0/3440872301 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fd580 0x7fed8c0fd9f0 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:47.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 --2- 172.21.15.105:0/3440872301 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c0fc6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:47.948 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- 172.21.15.105:0/3440872301 >> 172.21.15.105:0/3440872301 conn(0x7fed8c0f7a90 msgr2=0x7fed8c0f9ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- 172.21.15.105:0/3440872301 shutdown_connections 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- 172.21.15.105:0/3440872301 wait complete. 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 Processor -- start 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- start start 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c10cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed8c0fd580 0x7fed8c10d3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c10da30 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed915cc640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed8c10e120 con 0x7fed8c0fd580 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed8affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c10cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.941+0000 7fed8affd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c10cea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33442/0 (socket says 172.21.15.105:33442) 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed8affd640 1 -- 172.21.15.105:0/3402103243 learned_addr learned my addr 172.21.15.105:0/3402103243 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:52:47.949 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed8a7fc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed8c0fd580 0x7fed8c10d3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed8affd640 1 -- 172.21.15.105:0/3402103243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed8c0fd580 msgr2=0x7fed8c10d3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed8affd640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed8c0fd580 0x7fed8c10d3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed8affd640 1 -- 172.21.15.105:0/3402103243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed74005040 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed8affd640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c10cea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00db00 tx=0x7fed7c004e60 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed7c017070 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed7c00f040 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed8c10e3b0 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fed7c0039b0 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.942+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed8c10e950 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.943+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed8c0fb710 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.944+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fed7c003bf0 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.944+0000 7fed837fe640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fed78065430 0x7fed780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:47.950 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.944+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 4354+0+0 (secure 0 0 0) 0x7fed7c0824f0 con 0x7fed8c0fc2b0 2024-06-15T00:52:47.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.944+0000 7fed8a7fc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fed78065430 0x7fed780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:47.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.944+0000 7fed8a7fc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fed78065430 0x7fed780678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fed7400b020 tx=0x7fed74009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:47.951 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:47.948+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fed7c050ab0 con 0x7fed8c0fc2b0 2024-06-15T00:52:48.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.116+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7fed8c104ce0 con 0x7fed8c0fc2b0 2024-06-15T00:52:48.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.117+0000 7fed837fe640 1 -- 172.21.15.105:0/3402103243 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v39) v1 ==== 74+0+130 (secure 0 0 0) 0x7fed7c0540d0 con 0x7fed8c0fc2b0 2024-06-15T00:52:48.118 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fed78065430 msgr2=0x7fed780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fed78065430 0x7fed780678e0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 crypto rx=0x7fed7400b020 tx=0x7fed74009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 msgr2=0x7fed8c10cea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c10cea0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 crypto rx=0x7fed7c00db00 tx=0x7fed7c004e60 comp rx=0 tx=0).stop 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 shutdown_connections 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fed78065430 0x7fed780678e0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:48.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fed8c0fd580 0x7fed8c10d3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:48.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 --2- 172.21.15.105:0/3402103243 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fed8c0fc2b0 0x7fed8c10cea0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:48.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 >> 172.21.15.105:0/3402103243 conn(0x7fed8c0f7a90 msgr2=0x7fed8c0f91f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:48.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 shutdown_connections 2024-06-15T00:52:48.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:48.119+0000 7fed915cc640 1 -- 172.21.15.105:0/3402103243 wait complete. 2024-06-15T00:52:48.527 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:48 smithi143 ceph-mon[35860]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-06-15T00:52:48.527 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:48 smithi143 ceph-mon[35860]: osdmap e39: 6 total, 5 up, 6 in 2024-06-15T00:52:48.527 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:48 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:48.527 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:48 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:48.527 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:48 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3402103243' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-15T00:52:48.649 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:48 smithi105 ceph-mon[28877]: from='osd.5 ' entity='osd.5' cmd='[{"prefix": "osd crush create-or-move", "id": 5, "weight":0.0873, "args": ["host=smithi143", "root=default"]}]': finished 2024-06-15T00:52:48.649 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:48 smithi105 ceph-mon[28877]: osdmap e39: 6 total, 5 up, 6 in 2024-06-15T00:52:48.649 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:48 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:48.649 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:48 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:48.650 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:48 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3402103243' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-15T00:52:48.786 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":39,"num_osds":6,"num_up_osds":5,"osd_up_since":1718412743,"num_in_osds":6,"osd_in_since":1718412749,"num_remapped_pgs":0} 2024-06-15T00:52:49.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:49 smithi105 ceph-mon[28877]: purged_snaps scrub starts 2024-06-15T00:52:49.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:49 smithi105 ceph-mon[28877]: purged_snaps scrub ok 2024-06-15T00:52:49.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:49 smithi105 ceph-mon[28877]: osd.5 [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] boot 2024-06-15T00:52:49.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:49 smithi105 ceph-mon[28877]: osdmap e40: 6 total, 6 up, 6 in 2024-06-15T00:52:49.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:49 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:49.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:49 smithi105 ceph-mon[28877]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:49.613 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:49 smithi143 ceph-mon[35860]: purged_snaps scrub starts 2024-06-15T00:52:49.613 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:49 smithi143 ceph-mon[35860]: purged_snaps scrub ok 2024-06-15T00:52:49.613 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:49 smithi143 ceph-mon[35860]: osd.5 [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] boot 2024-06-15T00:52:49.613 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:49 smithi143 ceph-mon[35860]: osdmap e40: 6 total, 6 up, 6 in 2024-06-15T00:52:49.613 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:49 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd metadata", "id": 5}]: dispatch 2024-06-15T00:52:49.613 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:49 smithi143 ceph-mon[35860]: pgmap v130: 1 pgs: 1 active+clean; 577 KiB data, 1.4 GiB used, 446 GiB / 447 GiB avail 2024-06-15T00:52:49.787 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd stat -f json 2024-06-15T00:52:50.047 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:52:50.422 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: osdmap e41: 6 total, 6 up, 6 in 2024-06-15T00:52:50.422 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: Detected new or changed devices on smithi143 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: Adjusting osd_memory_target on smithi143 to 5312M 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:50.423 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:50.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:50.424 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: osdmap e41: 6 total, 6 up, 6 in 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: Detected new or changed devices on smithi143 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.3", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.4", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config rm", "who": "osd.5", "name": "osd_memory_target"}]: dispatch 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: Adjusting osd_memory_target on smithi143 to 5312M 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:50.886 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:50.887 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:52:50.887 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:52:50.887 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:52:50.887 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:52:51.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:51 smithi105 ceph-mon[28877]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:51.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:51 smithi143 ceph-mon[35860]: pgmap v132: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:51.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- 172.21.15.105:0/1069091555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 msgr2=0x7f58b80fddd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:51.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 --2- 172.21.15.105:0/1069091555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b80fddd0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 crypto rx=0x7f58b40048d0 tx=0x7f58b4009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.746 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- 172.21.15.105:0/1069091555 shutdown_connections 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 --2- 172.21.15.105:0/1069091555 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 0x7f58b80fa470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 --2- 172.21.15.105:0/1069091555 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b80fddd0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- 172.21.15.105:0/1069091555 >> 172.21.15.105:0/1069091555 conn(0x7f58b80f6100 msgr2=0x7f58b80f8530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- 172.21.15.105:0/1069091555 shutdown_connections 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- 172.21.15.105:0/1069091555 wait complete. 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 Processor -- start 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- start start 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b810f0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 0x7f58b810f630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b81103e0 con 0x7f58b80fd9c0 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58c0047640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58b8110550 con 0x7f58b80fe310 2024-06-15T00:52:51.747 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58bdde3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b810f0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58bdde3640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b810f0f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33460/0 (socket says 172.21.15.105:33460) 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.732+0000 7f58bdde3640 1 -- 172.21.15.105:0/774282634 learned_addr learned my addr 172.21.15.105:0/774282634 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58bd5e2640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 0x7f58b810f630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58bdde3640 1 -- 172.21.15.105:0/774282634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 msgr2=0x7f58b810f630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58bdde3640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 0x7f58b810f630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58bdde3640 1 -- 172.21.15.105:0/774282634 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58b4005040 con 0x7f58b80fd9c0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58bd5e2640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 0x7f58b810f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58bdde3640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b810f0f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009af0 tx=0x7f58b4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58b4038070 con 0x7f58b80fd9c0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f58b40096f0 con 0x7f58b80fd9c0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f58b81107d0 con 0x7f58b80fd9c0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f58b4032e60 con 0x7f58b80fd9c0 2024-06-15T00:52:51.748 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.733+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58b8110c90 con 0x7f58b80fd9c0 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.734+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58b80fcef0 con 0x7f58b80fd9c0 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.734+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f58b4030020 con 0x7f58b80fd9c0 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.735+0000 7f58ae7fc640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f58a40653d0 0x7f58a4067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.735+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f58b40a2e20 con 0x7f58b80fd9c0 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.735+0000 7f58bd5e2640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f58a40653d0 0x7f58a4067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.735+0000 7f58bd5e2640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f58a40653d0 0x7f58a4067880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58b810eea0 tx=0x7f58a8008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:51.749 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.739+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f58b4071390 con 0x7f58b80fd9c0 2024-06-15T00:52:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.903+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd stat", "format": "json"} v 0) v1 -- 0x7f58b80f9da0 con 0x7f58b80fd9c0 2024-06-15T00:52:51.904 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.904+0000 7f58ae7fc640 1 -- 172.21.15.105:0/774282634 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd stat", "format": "json"}]=0 v41) v1 ==== 74+0+130 (secure 0 0 0) 0x7f58b40749b0 con 0x7f58b80fd9c0 2024-06-15T00:52:51.904 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:52:51.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f58a40653d0 msgr2=0x7f58a4067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:51.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f58a40653d0 0x7f58a4067880 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 crypto rx=0x7f58b810eea0 tx=0x7f58a8008040 comp rx=0 tx=0).stop 2024-06-15T00:52:51.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 msgr2=0x7f58b810f0f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:51.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b810f0f0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 crypto rx=0x7f58b4009af0 tx=0x7f58b4009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 shutdown_connections 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f58a40653d0 0x7f58a4067880 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f58b80fe310 0x7f58b810f630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 --2- 172.21.15.105:0/774282634 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f58b80fd9c0 0x7f58b810f0f0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 >> 172.21.15.105:0/774282634 conn(0x7f58b80f6100 msgr2=0x7f58b80f7c40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 shutdown_connections 2024-06-15T00:52:51.907 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:51.906+0000 7f58c0047640 1 -- 172.21.15.105:0/774282634 wait complete. 2024-06-15T00:52:52.533 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:52 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/774282634' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-15T00:52:52.559 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"num_osds":6,"num_up_osds":6,"osd_up_since":1718412768,"num_in_osds":6,"osd_in_since":1718412749,"num_remapped_pgs":0} 2024-06-15T00:52:52.559 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-15T00:52:52.812 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:52:52.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:52 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/774282634' entity='client.admin' cmd=[{"prefix": "osd stat", "format": "json"}]: dispatch 2024-06-15T00:52:53.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:53 smithi105 ceph-mon[28877]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:53.885 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:53 smithi143 ceph-mon[35860]: pgmap v133: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:54.440 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 -- 172.21.15.105:0/200101238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 msgr2=0x7f5ac40fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 --2- 172.21.15.105:0/200101238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac40fc2f0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80048d0 tx=0x7f5ab8009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 -- 172.21.15.105:0/200101238 shutdown_connections 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 --2- 172.21.15.105:0/200101238 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac40fc2f0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 --2- 172.21.15.105:0/200101238 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ac41022c0 0x7f5ac4102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 -- 172.21.15.105:0/200101238 >> 172.21.15.105:0/200101238 conn(0x7f5ac40f7e50 msgr2=0x7f5ac40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 -- 172.21.15.105:0/200101238 shutdown_connections 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 -- 172.21.15.105:0/200101238 wait complete. 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 Processor -- start 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.437+0000 7f5ac983f640 1 -- start start 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac983f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ac41022c0 0x7f5ac4113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac983f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac41137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac983f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac4113db0 con 0x7f5ac4102c60 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac983f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ac410d1f0 con 0x7f5ac41022c0 2024-06-15T00:52:54.441 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac41137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac41137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33360/0 (socket says 172.21.15.105:33360) 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 -- 172.21.15.105:0/3934491803 learned_addr learned my addr 172.21.15.105:0/3934491803 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac2ffd640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ac41022c0 0x7f5ac4113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 -- 172.21.15.105:0/3934491803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ac41022c0 msgr2=0x7f5ac4113270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ac41022c0 0x7f5ac4113270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 -- 172.21.15.105:0/3934491803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ab8005040 con 0x7f5ac4102c60 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac27fc640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac41137b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80048a0 tx=0x7f5ab8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab8038070 con 0x7f5ac4102c60 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ac410d470 con 0x7f5ac4102c60 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.438+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ab802cef0 con 0x7f5ac4102c60 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.439+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f5ab8009640 con 0x7f5ac4102c60 2024-06-15T00:52:54.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.439+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ac410d9b0 con 0x7f5ac4102c60 2024-06-15T00:52:54.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.440+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ac40fda30 con 0x7f5ac4102c60 2024-06-15T00:52:54.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.440+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f5ab8030020 con 0x7f5ac4102c60 2024-06-15T00:52:54.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.440+0000 7f5aab7fe640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f5aac065430 0x7f5aac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:54.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.440+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f5ab80a28a0 con 0x7f5ac4102c60 2024-06-15T00:52:54.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.440+0000 7f5ac2ffd640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f5aac065430 0x7f5aac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:54.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.441+0000 7f5ac2ffd640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f5aac065430 0x7f5aac0678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4004ed0 tx=0x7f5ab400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:54.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.444+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f5ab8070dc0 con 0x7f5ac4102c60 2024-06-15T00:52:54.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.613+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5ac41092e0 con 0x7f5ac4102c60 2024-06-15T00:52:54.613 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.613+0000 7f5aab7fe640 1 -- 172.21.15.105:0/3934491803 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11108 (secure 0 0 0) 0x7f5ab80743e0 con 0x7f5ac4102c60 2024-06-15T00:52:54.614 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:52:54.614 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","created":"2024-06-15T00:47:25.567639+0000","modified":"2024-06-15T00:52:49.407127+0000","last_up_change":"2024-06-15T00:52:48.402136+0000","last_in_change":"2024-06-15T00:52:29.674193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-15T00:51:36.957070+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e146f885-f9f1-4f23-b08d-e3efaf314c9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6803","nonce":9359752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6805","nonce":9359752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6809","nonce":9359752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6807","nonce":9359752}]},"public_addr":"172.21.15.105:6803/9359752","cluster_addr":"172.21.15.105:6805/9359752","heartbeat_back_addr":"172.21.15.105:6809/9359752","heartbeat_front_addr":"172.21.15.105:6807/9359752","state":["exists","up"]},{"osd":1,"uuid":"f4e67024-91f3-4938-9f5f-df5ef15c20e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6811","nonce":3566937114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6813","nonce":3566937114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6817","nonce":3566937114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6815","nonce":3566937114}]},"public_addr":"172.21.15.105:6811/3566937114","cluster_addr":"172.21.15.105:6813/3566937114","heartbeat_back_addr":"172.21.15.105:6817/3566937114","heartbeat_front_addr":"172.21.15.105:6815/3566937114","state":["exists","up"]},{"osd":2,"uuid":"6200bbe6-b8df-4a68-92b1-10f9544a308e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6819","nonce":3804554438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6821","nonce":3804554438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6825","nonce":3804554438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6823","nonce":3804554438}]},"public_addr":"172.21.15.105:6819/3804554438","cluster_addr":"172.21.15.105:6821/3804554438","heartbeat_back_addr":"172.21.15.105:6825/3804554438","heartbeat_front_addr":"172.21.15.105:6823/3804554438","state":["exists","up"]},{"osd":3,"uuid":"b2ee8545-5b79-454a-97bc-a3975c9d1528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6801","nonce":2670827997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6803","nonce":2670827997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6807","nonce":2670827997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6805","nonce":2670827997}]},"public_addr":"172.21.15.143:6801/2670827997","cluster_addr":"172.21.15.143:6803/2670827997","heartbeat_back_addr":"172.21.15.143:6807/2670827997","heartbeat_front_addr":"172.21.15.143:6805/2670827997","state":["exists","up"]},{"osd":4,"uuid":"bda317b1-8168-4cb1-9593-c12cc95a00dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6809","nonce":3256896185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6811","nonce":3256896185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6815","nonce":3256896185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6813","nonce":3256896185}]},"public_addr":"172.21.15.143:6809/3256896185","cluster_addr":"172.21.15.143:6811/3256896185","heartbeat_back_addr":"172.21.15.143:6815/3256896185","heartbeat_front_addr":"172.21.15.143:6813/3256896185","state":["exists","up"]},{"osd":5,"uuid":"75426eb7-35b5-44f5-a6ce-ccb083273c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6817","nonce":4272939362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6819","nonce":4272939362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6823","nonce":4272939362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6821","nonce":4272939362}]},"public_addr":"172.21.15.143:6817/4272939362","cluster_addr":"172.21.15.143:6819/4272939362","heartbeat_back_addr":"172.21.15.143:6823/4272939362","heartbeat_front_addr":"172.21.15.143:6821/4272939362","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:50:38.621625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:06.071905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:33.459822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:56.499893+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:52:21.523442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:52:46.807662+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/3786409438":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/2692527002":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/1776775496":"2024-06-16T00:47:46.012311+0000","172.21.15.105:6800/2469050433":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6801/2469050433":"2024-06-16T00:48:22.614133+0000","172.21.15.105:0/3122292566":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/1185627900":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6801/135592780":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/2883738527":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/4259626155":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6800/135592780":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/3752234939":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6800/3568068422":"2024-06-16T00:49:24.657075+0000","172.21.15.105:6801/3568068422":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/45066059":"2024-06-16T00:49:24.657075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T00:52:54.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f5aac065430 msgr2=0x7f5aac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f5aac065430 0x7f5aac0678e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 crypto rx=0x7f5ab4004ed0 tx=0x7f5ab400b040 comp rx=0 tx=0).stop 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 msgr2=0x7f5ac41137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac41137b0 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 crypto rx=0x7f5ab80048a0 tx=0x7f5ab8009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 shutdown_connections 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f5aac065430 0x7f5aac0678e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f5ac4102c60 0x7f5ac41137b0 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 --2- 172.21.15.105:0/3934491803 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f5ac41022c0 0x7f5ac4113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 >> 172.21.15.105:0/3934491803 conn(0x7f5ac40f7e50 msgr2=0x7f5ac40f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.615+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 shutdown_connections 2024-06-15T00:52:54.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:54.616+0000 7f5ac983f640 1 -- 172.21.15.105:0/3934491803 wait complete. 2024-06-15T00:52:54.971 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:54 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3934491803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:52:55.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:54 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3934491803' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:52:55.397 INFO:tasks.cephadm.ceph_manager.ceph:[{'pool': 1, 'pool_name': '.mgr', 'create_time': '2024-06-15T00:51:36.957070+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 3, 'min_size': 2, 'crush_rule': 0, 'peering_crush_bucket_count': 0, 'peering_crush_bucket_target': 0, 'peering_crush_bucket_barrier': 0, 'peering_crush_bucket_mandatory_member': 2147483647, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_max': 32, 'pg_num_min': 1}, 'application_metadata': {'mgr': {}}}] 2024-06-15T00:52:55.397 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd pool get .mgr pg_num 2024-06-15T00:52:55.646 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:52:55.992 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:55.993 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:55 smithi105 ceph-mon[28877]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:56.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:52:56.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:55 smithi143 ceph-mon[35860]: pgmap v134: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:57.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 -- 172.21.15.105:0/3677526500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 msgr2=0x7f917410c910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:57.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 --2- 172.21.15.105:0/3677526500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f917410c910 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 crypto rx=0x7f91700048d0 tx=0x7f9170009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 -- 172.21.15.105:0/3677526500 shutdown_connections 2024-06-15T00:52:57.233 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 --2- 172.21.15.105:0/3677526500 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f917410c910 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 --2- 172.21.15.105:0/3677526500 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 0x7f91740fafc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 -- 172.21.15.105:0/3677526500 >> 172.21.15.105:0/3677526500 conn(0x7f91740709e0 msgr2=0x7f9174070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 -- 172.21.15.105:0/3677526500 shutdown_connections 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.213+0000 7f917b4ad640 1 -- 172.21.15.105:0/3677526500 wait complete. 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f917b4ad640 1 Processor -- start 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f917b4ad640 1 -- start start 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f917b4ad640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 0x7f91741195c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f917b4ad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f9174119b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f917b4ad640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917411a1e0 con 0x7f91740fb500 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f917b4ad640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9174113560 con 0x7f91740fabf0 2024-06-15T00:52:57.234 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f9174119b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f9174119b00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33390/0 (socket says 172.21.15.105:33390) 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 -- 172.21.15.105:0/863422043 learned_addr learned my addr 172.21.15.105:0/863422043 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 -- 172.21.15.105:0/863422043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 msgr2=0x7f91741195c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9179249640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 0x7f91741195c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 0x7f91741195c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 -- 172.21.15.105:0/863422043 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9170005040 con 0x7f91740fb500 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9179249640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 0x7f91741195c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.214+0000 7f9178a48640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f9174119b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9170004930 tx=0x7f9170009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.215+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9170038070 con 0x7f91740fb500 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.215+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9170032d40 con 0x7f91740fb500 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.215+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9174113790 con 0x7f91740fb500 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.215+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f917003dd70 con 0x7f91740fb500 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.215+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9174113c50 con 0x7f91740fb500 2024-06-15T00:52:57.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.216+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9174074980 con 0x7f91740fb500 2024-06-15T00:52:57.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.216+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9170030020 con 0x7f91740fb500 2024-06-15T00:52:57.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.217+0000 7f9165ffb640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9160065430 0x7f91600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:52:57.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.217+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f91700a3210 con 0x7f91740fb500 2024-06-15T00:52:57.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.217+0000 7f9179249640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9160065430 0x7f91600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:52:57.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.217+0000 7f9179249640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9160065430 0x7f91600678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9168006fd0 tx=0x7f9168008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:52:57.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.221+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9170071780 con 0x7f91740fb500 2024-06-15T00:52:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.387+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"} v 0) v1 -- 0x7f917404b510 con 0x7f91740fb500 2024-06-15T00:52:57.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.388+0000 7f9165ffb640 1 -- 172.21.15.105:0/863422043 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]=0 v41) v1 ==== 93+0+10 (secure 0 0 0) 0x7f9170074da0 con 0x7f91740fb500 2024-06-15T00:52:57.388 INFO:teuthology.orchestra.run.smithi105.stdout:pg_num: 1 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.389+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9160065430 msgr2=0x7f91600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9160065430 0x7f91600678e0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 crypto rx=0x7f9168006fd0 tx=0x7f9168008040 comp rx=0 tx=0).stop 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 msgr2=0x7f9174119b00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f9174119b00 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 crypto rx=0x7f9170004930 tx=0x7f9170009da0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 shutdown_connections 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9160065430 0x7f91600678e0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f91740fb500 0x7f9174119b00 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 --2- 172.21.15.105:0/863422043 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f91740fabf0 0x7f91741195c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:52:57.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 >> 172.21.15.105:0/863422043 conn(0x7f91740709e0 msgr2=0x7f91740fbf10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:52:57.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 shutdown_connections 2024-06-15T00:52:57.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:52:57.390+0000 7f917b4ad640 1 -- 172.21.15.105:0/863422043 wait complete. 2024-06-15T00:52:58.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:57 smithi105 ceph-mon[28877]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:58.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:57 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/863422043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-15T00:52:58.081 INFO:tasks.cephadm:Setting up client nodes... 2024-06-15T00:52:58.081 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.0 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-15T00:52:58.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:57 smithi143 ceph-mon[35860]: pgmap v135: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:52:58.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:57 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/863422043' entity='client.admin' cmd=[{"prefix": "osd pool get", "pool": ".mgr", "var": "pg_num"}]: dispatch 2024-06-15T00:52:58.334 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:52:59.839 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:52:59 smithi105 ceph-mon[28877]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:00.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.041+0000 7f1eac437640 1 -- 172.21.15.105:0/1851465738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fc290 msgr2=0x7f1ea40fc6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:00.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.041+0000 7f1eac437640 1 --2- 172.21.15.105:0/1851465738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fc290 0x7f1ea40fc6a0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 crypto rx=0x7f1ea00048d0 tx=0x7f1ea0009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- 172.21.15.105:0/1851465738 shutdown_connections 2024-06-15T00:53:00.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 --2- 172.21.15.105:0/1851465738 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ea40fd560 0x7f1ea40fd9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 --2- 172.21.15.105:0/1851465738 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fc290 0x7f1ea40fc6a0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- 172.21.15.105:0/1851465738 >> 172.21.15.105:0/1851465738 conn(0x7f1ea40f7a70 msgr2=0x7f1ea40f9e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- 172.21.15.105:0/1851465738 shutdown_connections 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- 172.21.15.105:0/1851465738 wait complete. 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 Processor -- start 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- start start 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ea40fc290 0x7f1ea410cf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 0x7f1ea410d440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea410e170 con 0x7f1ea40fd560 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.042+0000 7f1eac437640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ea410e2e0 con 0x7f1ea40fc290 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 0x7f1ea410d440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 0x7f1ea410d440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33404/0 (socket says 172.21.15.105:33404) 2024-06-15T00:53:00.044 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 -- 172.21.15.105:0/2492101398 learned_addr learned my addr 172.21.15.105:0/2492101398 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1eaa1d3640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ea40fc290 0x7f1ea410cf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 -- 172.21.15.105:0/2492101398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ea40fc290 msgr2=0x7f1ea410cf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ea40fc290 0x7f1ea410cf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 -- 172.21.15.105:0/2492101398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ea0005040 con 0x7f1ea40fd560 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1ea99d2640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 0x7f1ea410d440 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400b790 tx=0x7f1e940091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e9400c700 con 0x7f1ea40fd560 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e9400cd10 con 0x7f1ea40fd560 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ea410e5c0 con 0x7f1ea40fd560 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1e94013370 con 0x7f1ea40fd560 2024-06-15T00:53:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.043+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ea410eae0 con 0x7f1ea40fd560 2024-06-15T00:53:00.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.044+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ea40fb6f0 con 0x7f1ea40fd560 2024-06-15T00:53:00.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.045+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1e94012070 con 0x7f1ea40fd560 2024-06-15T00:53:00.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.045+0000 7f1e9affd640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1e90065430 0x7f1e900678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:00.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.045+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1e94081ec0 con 0x7f1ea40fd560 2024-06-15T00:53:00.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.045+0000 7f1eaa1d3640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1e90065430 0x7f1e900678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.045+0000 7f1eaa1d3640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1e90065430 0x7f1e900678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0000c00 tx=0x7f1ea0035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:00.049 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.049+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1e94050370 con 0x7f1ea40fd560 2024-06-15T00:53:00.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:52:59 smithi143 ceph-mon[35860]: pgmap v136: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:00.275 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.275+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7f1ea404b480 con 0x7f1ea40fd560 2024-06-15T00:53:00.280 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.280+0000 7f1e9affd640 1 -- 172.21.15.105:0/2492101398 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v14) v1 ==== 170+0+59 (secure 0 0 0) 0x7f1e94053990 con 0x7f1ea40fd560 2024-06-15T00:53:00.280 INFO:teuthology.orchestra.run.smithi105.stdout:[client.0] 2024-06-15T00:53:00.280 INFO:teuthology.orchestra.run.smithi105.stdout: key = AQDs5Wxme0yHEBAA2XOGCPQPULS7QWT6aegidA== 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1e90065430 msgr2=0x7f1e900678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1e90065430 0x7f1e900678e0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 crypto rx=0x7f1ea0000c00 tx=0x7f1ea0035040 comp rx=0 tx=0).stop 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 msgr2=0x7f1ea410d440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 0x7f1ea410d440 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 crypto rx=0x7f1e9400b790 tx=0x7f1e940091d0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 shutdown_connections 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1e90065430 0x7f1e900678e0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.282 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1ea40fd560 0x7f1ea410d440 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 --2- 172.21.15.105:0/2492101398 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1ea40fc290 0x7f1ea410cf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:00.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 >> 172.21.15.105:0/2492101398 conn(0x7f1ea40f7a70 msgr2=0x7f1ea40f9220 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:00.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 shutdown_connections 2024-06-15T00:53:00.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:00.282+0000 7f1eac437640 1 -- 172.21.15.105:0/2492101398 wait complete. 2024-06-15T00:53:00.877 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:00 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2492101398' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:53:00.877 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:00 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2492101398' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:53:00.901 DEBUG:teuthology.orchestra.run.smithi105:> set -ex 2024-06-15T00:53:00.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo dd of=/etc/ceph/ceph.client.0.keyring 2024-06-15T00:53:00.901 DEBUG:teuthology.orchestra.run.smithi105:> sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2024-06-15T00:53:00.938 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph auth get-or-create client.1 mon 'allow *' osd 'allow *' mds 'allow *' mgr 'allow *' 2024-06-15T00:53:01.036 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:00 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2492101398' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:53:01.037 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:00 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2492101398' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.0", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:53:01.223 INFO:teuthology.orchestra.run.smithi143.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi143/config 2024-06-15T00:53:02.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:02 smithi143 ceph-mon[35860]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 -- 172.21.15.143:0/4023891893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fd920 msgr2=0x7fc8f00fdd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/4023891893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fd920 0x7fc8f00fdd30 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 crypto rx=0x7fc8ec0048d0 tx=0x7fc8ec009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 -- 172.21.15.143:0/4023891893 shutdown_connections 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/4023891893 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8f00fe270 0x7fc8f00fa3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/4023891893 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fd920 0x7fc8f00fdd30 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 -- 172.21.15.143:0/4023891893 >> 172.21.15.143:0/4023891893 conn(0x7fc8f00f6100 msgr2=0x7fc8f00f8550 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.748+0000 7fc8f74cb640 1 -- 172.21.15.143:0/4023891893 shutdown_connections 2024-06-15T00:53:02.751 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 -- 172.21.15.143:0/4023891893 wait complete. 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 Processor -- start 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 -- start start 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8f00fd920 0x7fc8f010ce80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 0x7fc8f010d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f010da10 con 0x7fc8f00fd920 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f74cb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8f010e150 con 0x7fc8f00fe270 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f4a66640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 0x7fc8f010d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f4a66640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 0x7fc8f010d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.143:3300/0 says I am v2:172.21.15.143:41018/0 (socket says 172.21.15.143:41018) 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f4a66640 1 -- 172.21.15.143:0/1609460321 learned_addr learned my addr 172.21.15.143:0/1609460321 (peer_addr_for_me v2:172.21.15.143:0/0) 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.749+0000 7fc8f5267640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8f00fd920 0x7fc8f010ce80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8f4a66640 1 -- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8f00fd920 msgr2=0x7fc8f010ce80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8f4a66640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8f00fd920 0x7fc8f010ce80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.752 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8f4a66640 1 -- 172.21.15.143:0/1609460321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8ec005040 con 0x7fc8f00fe270 2024-06-15T00:53:02.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8f4a66640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 0x7fc8f010d3c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0002870 tx=0x7fc8e0004600 comp rx=0 tx=0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:02.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e0007c30 con 0x7fc8f00fe270 2024-06-15T00:53:02.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc8e000f040 con 0x7fc8f00fe270 2024-06-15T00:53:02.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8f010e3e0 con 0x7fc8f00fe270 2024-06-15T00:53:02.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc8e000da30 con 0x7fc8f00fe270 2024-06-15T00:53:02.753 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.750+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8f010e980 con 0x7fc8f00fe270 2024-06-15T00:53:02.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.751+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8f00f9dc0 con 0x7fc8f00fe270 2024-06-15T00:53:02.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.751+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc8e000db90 con 0x7fc8f00fe270 2024-06-15T00:53:02.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.752+0000 7fc8e5ffb640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc8dc065430 0x7fc8dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:02.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.752+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc8e0081b20 con 0x7fc8f00fe270 2024-06-15T00:53:02.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.752+0000 7fc8f5267640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc8dc065430 0x7fc8dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:02.754 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.753+0000 7fc8f5267640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc8dc065430 0x7fc8dc0678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc8f010cc10 tx=0x7fc8ec035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:02.756 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.756+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc8e004cdf0 con 0x7fc8f00fe270 2024-06-15T00:53:02.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:02 smithi105 ceph-mon[28877]: pgmap v137: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:02.980 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.979+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_command({"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]} v 0) v1 -- 0x7fc8f004b480 con 0x7fc8f00fe270 2024-06-15T00:53:02.985 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.984+0000 7fc8e5ffb640 1 -- 172.21.15.143:0/1609460321 <== mon.1 v2:172.21.15.143:3300/0 7 ==== mon_command_ack([{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]=0 v15) v1 ==== 170+0+59 (secure 0 0 0) 0x7fc8e0050100 con 0x7fc8f00fe270 2024-06-15T00:53:02.985 INFO:teuthology.orchestra.run.smithi143.stdout:[client.1] 2024-06-15T00:53:02.985 INFO:teuthology.orchestra.run.smithi143.stdout: key = AQDu5WxmbM+LOhAAKQoOn3s5PZfVYSmKAVJwvw== 2024-06-15T00:53:02.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.986+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc8dc065430 msgr2=0x7fc8dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:02.987 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.986+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc8dc065430 0x7fc8dc0678e0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 crypto rx=0x7fc8f010cc10 tx=0x7fc8ec035040 comp rx=0 tx=0).stop 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.986+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 msgr2=0x7fc8f010d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.986+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 0x7fc8f010d3c0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 crypto rx=0x7fc8e0002870 tx=0x7fc8e0004600 comp rx=0 tx=0).stop 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 shutdown_connections 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc8dc065430 0x7fc8dc0678e0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc8f00fe270 0x7fc8f010d3c0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 --2- 172.21.15.143:0/1609460321 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc8f00fd920 0x7fc8f010ce80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 >> 172.21.15.143:0/1609460321 conn(0x7fc8f00f6100 msgr2=0x7fc8f00f7870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 shutdown_connections 2024-06-15T00:53:02.988 INFO:teuthology.orchestra.run.smithi143.stderr:2024-06-15T00:53:02.987+0000 7fc8f74cb640 1 -- 172.21.15.143:0/1609460321 wait complete. 2024-06-15T00:53:03.566 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:03 smithi143 ceph-mon[35860]: from='client.? 172.21.15.143:0/1609460321' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:53:03.567 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:03 smithi143 ceph-mon[35860]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:53:03.567 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:03 smithi143 ceph-mon[35860]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:53:03.591 DEBUG:teuthology.orchestra.run.smithi143:> set -ex 2024-06-15T00:53:03.591 DEBUG:teuthology.orchestra.run.smithi143:> sudo dd of=/etc/ceph/ceph.client.1.keyring 2024-06-15T00:53:03.592 DEBUG:teuthology.orchestra.run.smithi143:> sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2024-06-15T00:53:03.629 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2024-06-15T00:53:03.630 INFO:tasks.cephadm.ceph_manager.ceph:waiting for mgr available 2024-06-15T00:53:03.630 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph mgr dump --format=json 2024-06-15T00:53:03.656 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:03 smithi105 ceph-mon[28877]: from='client.? 172.21.15.143:0/1609460321' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:53:03.656 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:03 smithi105 ceph-mon[28877]: from='client.? ' entity='client.admin' cmd=[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]: dispatch 2024-06-15T00:53:03.657 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:03 smithi105 ceph-mon[28877]: from='client.? ' entity='client.admin' cmd='[{"prefix": "auth get-or-create", "entity": "client.1", "caps": ["mon", "allow *", "osd", "allow *", "mds", "allow *", "mgr", "allow *"]}]': finished 2024-06-15T00:53:03.895 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:04.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:04 smithi143 ceph-mon[35860]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:04.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:04 smithi105 ceph-mon[28877]: pgmap v138: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:05.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 -- 172.21.15.105:0/3638398792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 msgr2=0x7fb4780fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:05.545 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 --2- 172.21.15.105:0/3638398792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4780fc2f0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 crypto rx=0x7fb4600048d0 tx=0x7fb460009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 -- 172.21.15.105:0/3638398792 shutdown_connections 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 --2- 172.21.15.105:0/3638398792 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4780fc2f0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 --2- 172.21.15.105:0/3638398792 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4781022c0 0x7fb478102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 -- 172.21.15.105:0/3638398792 >> 172.21.15.105:0/3638398792 conn(0x7fb4780f7e50 msgr2=0x7fb4780fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 -- 172.21.15.105:0/3638398792 shutdown_connections 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.539+0000 7fb47dae9640 1 -- 172.21.15.105:0/3638398792 wait complete. 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb47dae9640 1 Processor -- start 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb47dae9640 1 -- start start 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb47dae9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4781022c0 0x7fb478113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb47dae9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4781137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:05.546 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb47dae9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb478113db0 con 0x7fb478102c60 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb47dae9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb47810d1f0 con 0x7fb4781022c0 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4781137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4781137b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47778/0 (socket says 172.21.15.105:47778) 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 -- 172.21.15.105:0/936122096 learned_addr learned my addr 172.21.15.105:0/936122096 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb4777fe640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4781022c0 0x7fb478113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 -- 172.21.15.105:0/936122096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4781022c0 msgr2=0x7fb478113270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4781022c0 0x7fb478113270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 -- 172.21.15.105:0/936122096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb460005040 con 0x7fb478102c60 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.540+0000 7fb46ffff640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4781137b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb4600048a0 tx=0x7fb460009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.541+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb460038070 con 0x7fb478102c60 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.541+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb46002cef0 con 0x7fb478102c60 2024-06-15T00:53:05.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.541+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb47810d470 con 0x7fb478102c60 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.541+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb460009640 con 0x7fb478102c60 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.541+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb47810d9b0 con 0x7fb478102c60 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.542+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4780fda30 con 0x7fb478102c60 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.542+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb460030020 con 0x7fb478102c60 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.542+0000 7fb474ff9640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb464065430 0x7fb4640678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.542+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb4600a28e0 con 0x7fb478102c60 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.542+0000 7fb4777fe640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb464065430 0x7fb4640678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:05.548 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.543+0000 7fb4777fe640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb464065430 0x7fb4640678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb468004ed0 tx=0x7fb46800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:05.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.547+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb460070d90 con 0x7fb478102c60 2024-06-15T00:53:05.753 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.752+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7fb4781092e0 con 0x7fb478102c60 2024-06-15T00:53:05.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.755+0000 7fb474ff9640 1 -- 172.21.15.105:0/936122096 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v19) v1 ==== 74+0+161456 (secure 0 0 0) 0x7fb4600743b0 con 0x7fb478102c60 2024-06-15T00:53:05.761 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:05.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb464065430 msgr2=0x7fb4640678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:05.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb464065430 0x7fb4640678e0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 crypto rx=0x7fb468004ed0 tx=0x7fb46800b040 comp rx=0 tx=0).stop 2024-06-15T00:53:05.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 msgr2=0x7fb4781137b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:05.763 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4781137b0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 crypto rx=0x7fb4600048a0 tx=0x7fb460009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 shutdown_connections 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb464065430 0x7fb4640678e0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb478102c60 0x7fb4781137b0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 --2- 172.21.15.105:0/936122096 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb4781022c0 0x7fb478113270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 >> 172.21.15.105:0/936122096 conn(0x7fb4780f7e50 msgr2=0x7fb4780f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 shutdown_connections 2024-06-15T00:53:05.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:05.758+0000 7fb47dae9640 1 -- 172.21.15.105:0/936122096 wait complete. 2024-06-15T00:53:06.526 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":19,"flags":0,"active_gid":14214,"active_name":"smithi105.oorfym","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6800","nonce":3392458267},{"type":"v1","addr":"172.21.15.105:6801","nonce":3392458267}]},"active_addr":"172.21.15.105:6801/3392458267","active_change":"2024-06-15T00:49:24.657214+0000","active_mgr_features":4540138320759226367,"available":true,"standbys":[{"gid":14232,"name":"smithi143.hhtfap","mgr_features":4540138320759226367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["cephadm","dashboard","iostat","nfs","prometheus","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"upmap","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"agent_down_multiplier":{"name":"agent_down_multiplier","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"","max":"","enum_allowed":[],"desc":"Multiplied by agent refresh rate to calculate how long agent must not report before being marked down","long_desc":"","tags":[],"see_also":[]},"agent_refresh_rate":{"name":"agent_refresh_rate","type":"secs","level":"advanced","flags":0,"default_value":"20","min":"","max":"","enum_allowed":[],"desc":"How often agent on each host will try to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"agent_starting_port":{"name":"agent_starting_port","type":"int","level":"advanced","flags":0,"default_value":"4721","min":"","max":"","enum_allowed":[],"desc":"First port agent will try to bind to (will also try up to next 1000 subsequent ports if blocked)","long_desc":"","tags":[],"see_also":[]},"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"autotune_interval":{"name":"autotune_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to autotune daemon memory","long_desc":"","tags":[],"see_also":[]},"autotune_memory_target_ratio":{"name":"autotune_memory_target_ratio","type":"float","level":"advanced","flags":0,"default_value":"0.7","min":"","max":"","enum_allowed":[],"desc":"ratio of total system memory to divide amongst autotuned daemons","long_desc":"","tags":[],"see_also":[]},"cgroups_split":{"name":"cgroups_split","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Pass --cgroups=split when cephadm creates containers (currently podman only)","long_desc":"","tags":[],"see_also":[]},"config_checks_enabled":{"name":"config_checks_enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable or disable the cephadm configuration analysis","long_desc":"","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.25.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:9.4.7","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_haproxy":{"name":"container_image_haproxy","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/haproxy:2.3","min":"","max":"","enum_allowed":[],"desc":"HAproxy container image","long_desc":"","tags":[],"see_also":[]},"container_image_keepalived":{"name":"container_image_keepalived","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/keepalived:2.1.5","min":"","max":"","enum_allowed":[],"desc":"Keepalived container image","long_desc":"","tags":[],"see_also":[]},"container_image_loki":{"name":"container_image_loki","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/loki:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Loki container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v1.5.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.43.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_promtail":{"name":"container_image_promtail","type":"str","level":"advanced","flags":0,"default_value":"docker.io/grafana/promtail:2.4.0","min":"","max":"","enum_allowed":[],"desc":"Promtail container image","long_desc":"","tags":[],"see_also":[]},"container_image_snmp_gateway":{"name":"container_image_snmp_gateway","type":"str","level":"advanced","flags":0,"default_value":"docker.io/maxwo/snmp-notifier:v1.2.1","min":"","max":"","enum_allowed":[],"desc":"SNMP Gateway container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"default_cephadm_command_timeout":{"name":"default_cephadm_command_timeout","type":"int","level":"advanced","flags":0,"default_value":"900","min":"","max":"","enum_allowed":[],"desc":"Default timeout applied to cephadm commands run directly on the host (in seconds)","long_desc":"","tags":[],"see_also":[]},"default_registry":{"name":"default_registry","type":"str","level":"advanced","flags":0,"default_value":"docker.io","min":"","max":"","enum_allowed":[],"desc":"Search-registry to which we should normalize unqualified image names. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"device_enhanced_scan":{"name":"device_enhanced_scan","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use libstoragemgmt during device scans","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"inventory_list_all":{"name":"inventory_list_all","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Whether ceph-volume inventory should report more devices (mostly mappers (LVs / mpaths), partitions...)","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_refresh_metadata":{"name":"log_refresh_metadata","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Log all refresh metadata. Includes daemon, device, and host info collected regularly. Only has effect if logging at debug level","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf_hosts":{"name":"manage_etc_ceph_ceph_conf_hosts","type":"str","level":"advanced","flags":0,"default_value":"*","min":"","max":"","enum_allowed":[],"desc":"PlacementSpec describing on which hosts to manage /etc/ceph/ceph.conf","long_desc":"","tags":[],"see_also":[]},"max_count_per_host":{"name":"max_count_per_host","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of daemons per service per host","long_desc":"","tags":[],"see_also":[]},"max_osd_draining_count":{"name":"max_osd_draining_count","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"max number of osds that will be drained simultaneously when osds are removed","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_insecure":{"name":"registry_insecure","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Registry is to be considered insecure (no TLS available). Only for development purposes.","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository password. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Registry url for login purposes. This is not the default registry","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Custom repository username. Only used for logging into a registry.","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_agent":{"name":"use_agent","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Use cephadm agent on each host to gather and send metadata","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_CEPHFS":{"name":"FEATURE_TOGGLE_CEPHFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_DASHBOARD":{"name":"FEATURE_TOGGLE_DASHBOARD","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_ISCSI":{"name":"FEATURE_TOGGLE_ISCSI","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_MIRRORING":{"name":"FEATURE_TOGGLE_MIRRORING","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_NFS":{"name":"FEATURE_TOGGLE_NFS","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RBD":{"name":"FEATURE_TOGGLE_RBD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_RGW":{"name":"FEATURE_TOGGLE_RGW","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISSUE_TRACKER_API_KEY":{"name":"ISSUE_TRACKER_API_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"UNSAFE_TLS_v1_2":{"name":"UNSAFE_TLS_v1_2","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"cross_origin_url":{"name":"cross_origin_url","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"redirect_resolve_ip_addr":{"name":"redirect_resolve_ip_addr","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"int","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"How big batches of data points should be when sending to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"InfluxDB database name. You will need to create this database and grant write privileges to the configured username or the username must have admin privileges to create it.","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server hostname","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"30","min":"5","max":"","enum_allowed":[],"desc":"Time between reports to InfluxDB. Default 30 seconds.","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"password of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"int","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"InfluxDB server port","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"Use https connection for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"int","level":"advanced","flags":0,"default_value":"5","min":"1","max":"32","enum_allowed":[],"desc":"How many worker threads should be spawned for sending data to InfluxDB.","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"username of InfluxDB server user","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"Verify https cert for InfluxDB server. Use \"true\" or \"false\".","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"k8sevents","can_run":false,"error_string":"No module named 'kubernetes.client.models.v1_event'","module_options":{}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"what failure domain we should separate data replicas across.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"value to set min_size to (unchanged from Ceph's default if this option is not set)","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"which CRUSH subtree type the module should create a pool for.","tags":[],"see_also":[]}}},{"name":"mds_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"mirroring","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"nfs","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_perf_query","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threshold":{"name":"threshold","type":"float","level":"advanced","flags":0,"default_value":"3.0","min":"1.0","max":"","enum_allowed":[],"desc":"scaling threshold","long_desc":"The factor by which the `NEW PG_NUM` must vary from the current`PG_NUM` before being accepted. Cannot be less than 1.0","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"allow_pg_recovery_event":{"name":"allow_pg_recovery_event","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow the module to show pg recovery progress","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how long the module is going to sleep","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"cache":{"name":"cache","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"float","level":"advanced","flags":0,"default_value":"15.0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"the IPv4 or IPv6 address on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":1,"default_value":"9283","min":"","max":"","enum_allowed":[],"desc":"the port on which the module listens for HTTP requests","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"log","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":1,"default_value":"default","min":"","max":"","enum_allowed":["default","error"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":1,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"enable_auth":{"name":"enable_auth","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rgw","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rook","can_run":true,"error_string":"","module_options":{"drive_group_interval":{"name":"drive_group_interval","type":"float","level":"advanced","flags":0,"default_value":"300.0","min":"","max":"","enum_allowed":[],"desc":"interval in seconds between re-application of applied drive_groups","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"storage_class":{"name":"storage_class","type":"str","level":"advanced","flags":0,"default_value":"local","min":"","max":"","enum_allowed":[],"desc":"storage class name for LSO-discovered PVs","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption7":{"name":"rwoption7","type":"int","level":"advanced","flags":0,"default_value":"","min":"1","max":"42","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"snap_schedule","can_run":true,"error_string":"","module_options":{"allow_m_granularity":{"name":"allow_m_granularity","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow minute scheduled snapshots","long_desc":"","tags":[],"see_also":[]},"dump_on_update":{"name":"dump_on_update","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"dump database to debug log on update","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"stats","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share metadata about Ceph daemon crashes (version, stack straces, etc)","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share device health metrics (e.g., SMART data, minus potentially identifying info like serial numbers)","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share a user-provided description and/or contact email for the cluster","long_desc":"","tags":[],"see_also":[]},"channel_perf":{"name":"channel_perf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Share various performance metrics of a cluster","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"uint","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{"dashboard":"https://172.21.15.105:8443/","prometheus":"http://172.21.15.105:9283/"},"always_on_modules":{"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"pacific":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"quincy":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"]},"last_failure_osd_epoch":5,"active_clients":[{"name":"devicehealth","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":4102562770}]},{"name":"libcephsqlite","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":2496428186}]},{"name":"rbd_support","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1985390595}]},{"name":"volumes","addrvec":[{"type":"v2","addr":"172.21.15.105:0","nonce":1689478630}]}]} 2024-06-15T00:53:06.529 INFO:tasks.cephadm.ceph_manager.ceph:mgr available! 2024-06-15T00:53:06.530 INFO:tasks.cephadm.ceph_manager.ceph:waiting for all up 2024-06-15T00:53:06.530 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-15T00:53:06.589 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:06 smithi105 ceph-mon[28877]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:06.589 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:06 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/936122096' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-15T00:53:06.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:06 smithi143 ceph-mon[35860]: pgmap v139: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:06.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:06 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/936122096' entity='client.admin' cmd=[{"prefix": "mgr dump", "format": "json"}]: dispatch 2024-06-15T00:53:06.818 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:08.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 -- 172.21.15.105:0/1418827447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 msgr2=0x7fdc200fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:08.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1418827447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc200fc2d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 crypto rx=0x7fdc100048d0 tx=0x7fdc10009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 -- 172.21.15.105:0/1418827447 shutdown_connections 2024-06-15T00:53:08.477 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1418827447 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc200fc2d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1418827447 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc201022a0 0x7fdc20102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 -- 172.21.15.105:0/1418827447 >> 172.21.15.105:0/1418827447 conn(0x7fdc200f7e50 msgr2=0x7fdc200fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 -- 172.21.15.105:0/1418827447 shutdown_connections 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 -- 172.21.15.105:0/1418827447 wait complete. 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 Processor -- start 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.472+0000 7fdc27be0640 1 -- start start 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc27be0640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc201022a0 0x7fdc20112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc27be0640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc201134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc27be0640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc20113b50 con 0x7fdc20102c40 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc27be0640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc2010cf00 con 0x7fdc201022a0 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc201134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:08.478 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc201134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47802/0 (socket says 172.21.15.105:47802) 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 -- 172.21.15.105:0/1923096512 learned_addr learned my addr 172.21.15.105:0/1923096512 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2597c640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc201022a0 0x7fdc20112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 -- 172.21.15.105:0/1923096512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc201022a0 msgr2=0x7fdc20112f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc201022a0 0x7fdc20112f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 -- 172.21.15.105:0/1923096512 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc10005040 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc2517b640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc201134c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc100048a0 tx=0x7fdc10009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc10038070 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc10032d30 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc2010d180 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdc10009680 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.473+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc2010d640 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.474+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc200fda10 con 0x7fdc20102c40 2024-06-15T00:53:08.479 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.475+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdc10030020 con 0x7fdc20102c40 2024-06-15T00:53:08.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.475+0000 7fdc167fc640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdc0c065430 0x7fdc0c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:08.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.475+0000 7fdc2597c640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdc0c065430 0x7fdc0c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:08.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.475+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdc100a2300 con 0x7fdc20102c40 2024-06-15T00:53:08.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.476+0000 7fdc2597c640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdc0c065430 0x7fdc0c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00a900 tx=0x7fdc1c008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:08.480 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.479+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdc100707b0 con 0x7fdc20102c40 2024-06-15T00:53:08.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:08 smithi143 ceph-mon[35860]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:08.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.644+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fdc20109380 con 0x7fdc20102c40 2024-06-15T00:53:08.644 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:08 smithi105 ceph-mon[28877]: pgmap v140: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:08.644 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.645+0000 7fdc167fc640 1 -- 172.21.15.105:0/1923096512 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11108 (secure 0 0 0) 0x7fdc10073dd0 con 0x7fdc20102c40 2024-06-15T00:53:08.645 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:08.645 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","created":"2024-06-15T00:47:25.567639+0000","modified":"2024-06-15T00:52:49.407127+0000","last_up_change":"2024-06-15T00:52:48.402136+0000","last_in_change":"2024-06-15T00:52:29.674193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-15T00:51:36.957070+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e146f885-f9f1-4f23-b08d-e3efaf314c9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6803","nonce":9359752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6805","nonce":9359752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6809","nonce":9359752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6807","nonce":9359752}]},"public_addr":"172.21.15.105:6803/9359752","cluster_addr":"172.21.15.105:6805/9359752","heartbeat_back_addr":"172.21.15.105:6809/9359752","heartbeat_front_addr":"172.21.15.105:6807/9359752","state":["exists","up"]},{"osd":1,"uuid":"f4e67024-91f3-4938-9f5f-df5ef15c20e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6811","nonce":3566937114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6813","nonce":3566937114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6817","nonce":3566937114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6815","nonce":3566937114}]},"public_addr":"172.21.15.105:6811/3566937114","cluster_addr":"172.21.15.105:6813/3566937114","heartbeat_back_addr":"172.21.15.105:6817/3566937114","heartbeat_front_addr":"172.21.15.105:6815/3566937114","state":["exists","up"]},{"osd":2,"uuid":"6200bbe6-b8df-4a68-92b1-10f9544a308e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6819","nonce":3804554438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6821","nonce":3804554438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6825","nonce":3804554438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6823","nonce":3804554438}]},"public_addr":"172.21.15.105:6819/3804554438","cluster_addr":"172.21.15.105:6821/3804554438","heartbeat_back_addr":"172.21.15.105:6825/3804554438","heartbeat_front_addr":"172.21.15.105:6823/3804554438","state":["exists","up"]},{"osd":3,"uuid":"b2ee8545-5b79-454a-97bc-a3975c9d1528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6801","nonce":2670827997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6803","nonce":2670827997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6807","nonce":2670827997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6805","nonce":2670827997}]},"public_addr":"172.21.15.143:6801/2670827997","cluster_addr":"172.21.15.143:6803/2670827997","heartbeat_back_addr":"172.21.15.143:6807/2670827997","heartbeat_front_addr":"172.21.15.143:6805/2670827997","state":["exists","up"]},{"osd":4,"uuid":"bda317b1-8168-4cb1-9593-c12cc95a00dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6809","nonce":3256896185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6811","nonce":3256896185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6815","nonce":3256896185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6813","nonce":3256896185}]},"public_addr":"172.21.15.143:6809/3256896185","cluster_addr":"172.21.15.143:6811/3256896185","heartbeat_back_addr":"172.21.15.143:6815/3256896185","heartbeat_front_addr":"172.21.15.143:6813/3256896185","state":["exists","up"]},{"osd":5,"uuid":"75426eb7-35b5-44f5-a6ce-ccb083273c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6817","nonce":4272939362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6819","nonce":4272939362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6823","nonce":4272939362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6821","nonce":4272939362}]},"public_addr":"172.21.15.143:6817/4272939362","cluster_addr":"172.21.15.143:6819/4272939362","heartbeat_back_addr":"172.21.15.143:6823/4272939362","heartbeat_front_addr":"172.21.15.143:6821/4272939362","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:50:38.621625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:06.071905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:33.459822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:56.499893+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:52:21.523442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:52:46.807662+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/3786409438":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/2692527002":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/1776775496":"2024-06-16T00:47:46.012311+0000","172.21.15.105:6800/2469050433":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6801/2469050433":"2024-06-16T00:48:22.614133+0000","172.21.15.105:0/3122292566":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/1185627900":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6801/135592780":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/2883738527":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/4259626155":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6800/135592780":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/3752234939":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6800/3568068422":"2024-06-16T00:49:24.657075+0000","172.21.15.105:6801/3568068422":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/45066059":"2024-06-16T00:49:24.657075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T00:53:08.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdc0c065430 msgr2=0x7fdc0c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:08.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdc0c065430 0x7fdc0c0678e0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 crypto rx=0x7fdc1c00a900 tx=0x7fdc1c008040 comp rx=0 tx=0).stop 2024-06-15T00:53:08.647 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 msgr2=0x7fdc201134c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc201134c0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 crypto rx=0x7fdc100048a0 tx=0x7fdc10009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 shutdown_connections 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdc0c065430 0x7fdc0c0678e0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdc20102c40 0x7fdc201134c0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 --2- 172.21.15.105:0/1923096512 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdc201022a0 0x7fdc20112f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 >> 172.21.15.105:0/1923096512 conn(0x7fdc200f7e50 msgr2=0x7fdc200f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 shutdown_connections 2024-06-15T00:53:08.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:08.647+0000 7fdc27be0640 1 -- 172.21.15.105:0/1923096512 wait complete. 2024-06-15T00:53:09.308 INFO:tasks.cephadm.ceph_manager.ceph:all up! 2024-06-15T00:53:09.308 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd dump --format=json 2024-06-15T00:53:09.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:09 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1923096512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:53:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:09.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:09 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1923096512' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:53:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:10 smithi143 ceph-mon[35860]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:10.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:10.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:10 smithi105 ceph-mon[28877]: pgmap v141: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:10.713 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 -- 172.21.15.105:0/2821127385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 msgr2=0x7f3a04065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 --2- 172.21.15.105:0/2821127385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a04065470 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0048d0 tx=0x7f39fc009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 -- 172.21.15.105:0/2821127385 shutdown_connections 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 --2- 172.21.15.105:0/2821127385 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a040fdeb0 0x7f3a040fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 --2- 172.21.15.105:0/2821127385 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a04065470 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 -- 172.21.15.105:0/2821127385 >> 172.21.15.105:0/2821127385 conn(0x7f3a04072110 msgr2=0x7f3a04072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.170+0000 7f3a09137640 1 -- 172.21.15.105:0/2821127385 shutdown_connections 2024-06-15T00:53:11.175 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 -- 172.21.15.105:0/2821127385 wait complete. 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 Processor -- start 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 -- start start 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a040fb2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a040fdeb0 0x7f3a040fb800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040fbdd0 con 0x7f3a040650a0 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a09137640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a040fbf40 con 0x7f3a040fdeb0 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a02d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a040fb2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a02d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a040fb2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47820/0 (socket says 172.21.15.105:47820) 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.171+0000 7f3a02d9d640 1 -- 172.21.15.105:0/922001884 learned_addr learned my addr 172.21.15.105:0/922001884 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f3a02d9d640 1 -- 172.21.15.105:0/922001884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a040fdeb0 msgr2=0x7f3a040fb800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f3a02d9d640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a040fdeb0 0x7f3a040fb800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f3a02d9d640 1 -- 172.21.15.105:0/922001884 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39fc005040 con 0x7f3a040650a0 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f3a02d9d640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a040fb2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0048a0 tx=0x7f39fc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:11.176 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39fc038070 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a041192e0 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39fc032d30 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f39fc009680 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.172+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a04119660 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.173+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a040fda50 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.174+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f39fc030020 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.174+0000 7f39eb7fe640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f39ec065430 0x7f39ec0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.174+0000 7f3a0259c640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f39ec065430 0x7f39ec0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.174+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f39fc0a2300 con 0x7f3a040650a0 2024-06-15T00:53:11.177 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.175+0000 7f3a0259c640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f39ec065430 0x7f39ec0678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a040fcc00 tx=0x7f39f8006d20 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:11.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.179+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f39fc0707b0 con 0x7f3a040650a0 2024-06-15T00:53:11.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.346+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3a04072c20 con 0x7f3a040650a0 2024-06-15T00:53:11.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.347+0000 7f39eb7fe640 1 -- 172.21.15.105:0/922001884 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+11108 (secure 0 0 0) 0x7f39fc073dd0 con 0x7f3a040650a0 2024-06-15T00:53:11.347 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:11.347 INFO:teuthology.orchestra.run.smithi105.stdout:{"epoch":41,"fsid":"bd0fff6c-2ab0-11ef-bc9d-c7b262605968","created":"2024-06-15T00:47:25.567639+0000","modified":"2024-06-15T00:52:49.407127+0000","last_up_change":"2024-06-15T00:52:48.402136+0000","last_in_change":"2024-06-15T00:52:29.674193+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":14,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":6,"require_min_compat_client":"luminous","min_compat_client":"jewel","require_osd_release":"quincy","pools":[{"pool":1,"pool_name":".mgr","create_time":"2024-06-15T00:51:36.957070+0000","flags":1,"flags_names":"hashpspool","type":1,"size":3,"min_size":2,"crush_rule":0,"peering_crush_bucket_count":0,"peering_crush_bucket_target":0,"peering_crush_bucket_barrier":0,"peering_crush_bucket_mandatory_member":2147483647,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_max":32,"pg_num_min":1},"application_metadata":{"mgr":{}}}],"osds":[{"osd":0,"uuid":"e146f885-f9f1-4f23-b08d-e3efaf314c9b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":9,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6802","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6803","nonce":9359752}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6804","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6805","nonce":9359752}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6808","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6809","nonce":9359752}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6806","nonce":9359752},{"type":"v1","addr":"172.21.15.105:6807","nonce":9359752}]},"public_addr":"172.21.15.105:6803/9359752","cluster_addr":"172.21.15.105:6805/9359752","heartbeat_back_addr":"172.21.15.105:6809/9359752","heartbeat_front_addr":"172.21.15.105:6807/9359752","state":["exists","up"]},{"osd":1,"uuid":"f4e67024-91f3-4938-9f5f-df5ef15c20e2","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":14,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6810","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6811","nonce":3566937114}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6812","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6813","nonce":3566937114}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6816","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6817","nonce":3566937114}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6814","nonce":3566937114},{"type":"v1","addr":"172.21.15.105:6815","nonce":3566937114}]},"public_addr":"172.21.15.105:6811/3566937114","cluster_addr":"172.21.15.105:6813/3566937114","heartbeat_back_addr":"172.21.15.105:6817/3566937114","heartbeat_front_addr":"172.21.15.105:6815/3566937114","state":["exists","up"]},{"osd":2,"uuid":"6200bbe6-b8df-4a68-92b1-10f9544a308e","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":19,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6818","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6819","nonce":3804554438}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6820","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6821","nonce":3804554438}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6824","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6825","nonce":3804554438}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.105:6822","nonce":3804554438},{"type":"v1","addr":"172.21.15.105:6823","nonce":3804554438}]},"public_addr":"172.21.15.105:6819/3804554438","cluster_addr":"172.21.15.105:6821/3804554438","heartbeat_back_addr":"172.21.15.105:6825/3804554438","heartbeat_front_addr":"172.21.15.105:6823/3804554438","state":["exists","up"]},{"osd":3,"uuid":"b2ee8545-5b79-454a-97bc-a3975c9d1528","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":27,"up_thru":34,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6800","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6801","nonce":2670827997}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6802","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6803","nonce":2670827997}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6806","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6807","nonce":2670827997}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6804","nonce":2670827997},{"type":"v1","addr":"172.21.15.143:6805","nonce":2670827997}]},"public_addr":"172.21.15.143:6801/2670827997","cluster_addr":"172.21.15.143:6803/2670827997","heartbeat_back_addr":"172.21.15.143:6807/2670827997","heartbeat_front_addr":"172.21.15.143:6805/2670827997","state":["exists","up"]},{"osd":4,"uuid":"bda317b1-8168-4cb1-9593-c12cc95a00dd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":35,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6808","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6809","nonce":3256896185}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6810","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6811","nonce":3256896185}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6814","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6815","nonce":3256896185}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6812","nonce":3256896185},{"type":"v1","addr":"172.21.15.143:6813","nonce":3256896185}]},"public_addr":"172.21.15.143:6809/3256896185","cluster_addr":"172.21.15.143:6811/3256896185","heartbeat_back_addr":"172.21.15.143:6815/3256896185","heartbeat_front_addr":"172.21.15.143:6813/3256896185","state":["exists","up"]},{"osd":5,"uuid":"75426eb7-35b5-44f5-a6ce-ccb083273c69","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":40,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6816","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6817","nonce":4272939362}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6818","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6819","nonce":4272939362}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6822","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6823","nonce":4272939362}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.143:6820","nonce":4272939362},{"type":"v1","addr":"172.21.15.143:6821","nonce":4272939362}]},"public_addr":"172.21.15.143:6817/4272939362","cluster_addr":"172.21.15.143:6819/4272939362","heartbeat_back_addr":"172.21.15.143:6823/4272939362","heartbeat_front_addr":"172.21.15.143:6821/4272939362","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:50:38.621625+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:06.071905+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:33.459822+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:51:56.499893+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:52:21.523442+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138320759226367,"old_weight":0,"last_purged_snaps_scrub":"2024-06-15T00:52:46.807662+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blocklist":{"172.21.15.105:0/3786409438":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/2692527002":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/1776775496":"2024-06-16T00:47:46.012311+0000","172.21.15.105:6800/2469050433":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6801/2469050433":"2024-06-16T00:48:22.614133+0000","172.21.15.105:0/3122292566":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/1185627900":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6801/135592780":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/2883738527":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/4259626155":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6800/135592780":"2024-06-16T00:47:46.012311+0000","172.21.15.105:0/3752234939":"2024-06-16T00:48:22.614133+0000","172.21.15.105:6800/3568068422":"2024-06-16T00:49:24.657075+0000","172.21.15.105:6801/3568068422":"2024-06-16T00:49:24.657075+0000","172.21.15.105:0/45066059":"2024-06-16T00:49:24.657075+0000"},"range_blocklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{},"stretch_mode":{"stretch_mode_enabled":false,"stretch_bucket_count":0,"degraded_stretch_mode":0,"recovering_stretch_mode":0,"stretch_mode_bucket":0}} 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f39ec065430 msgr2=0x7f39ec0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f39ec065430 0x7f39ec0678e0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 crypto rx=0x7f3a040fcc00 tx=0x7f39f8006d20 comp rx=0 tx=0).stop 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 msgr2=0x7f3a040fb2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a040fb2c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 crypto rx=0x7f39fc0048a0 tx=0x7f39fc009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 shutdown_connections 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f39ec065430 0x7f39ec0678e0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3a040fdeb0 0x7f3a040fb800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.349 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 --2- 172.21.15.105:0/922001884 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3a040650a0 0x7f3a040fb2c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.348+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 >> 172.21.15.105:0/922001884 conn(0x7f3a04072110 msgr2=0x7f3a040f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.349+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 shutdown_connections 2024-06-15T00:53:11.350 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:11.349+0000 7f3a09137640 1 -- 172.21.15.105:0/922001884 wait complete. 2024-06-15T00:53:12.062 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph tell osd.0 flush_pg_stats 2024-06-15T00:53:12.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph tell osd.1 flush_pg_stats 2024-06-15T00:53:12.063 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph tell osd.2 flush_pg_stats 2024-06-15T00:53:12.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph tell osd.3 flush_pg_stats 2024-06-15T00:53:12.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph tell osd.4 flush_pg_stats 2024-06-15T00:53:12.064 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph tell osd.5 flush_pg_stats 2024-06-15T00:53:12.370 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:12.373 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:12.574 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:12 smithi105 ceph-mon[28877]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:12.574 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:12 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/922001884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:53:12.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:12 smithi143 ceph-mon[35860]: pgmap v142: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:12.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:12 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/922001884' entity='client.admin' cmd=[{"prefix": "osd dump", "format": "json"}]: dispatch 2024-06-15T00:53:14.308 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/3366262871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d4102c60 msgr2=0x7fa8d40fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/3366262871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d4102c60 0x7fa8d40fc2f0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 crypto rx=0x7fa8c40048d0 tx=0x7fa8c4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/3366262871 shutdown_connections 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/3366262871 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d4102c60 0x7fa8d40fc2f0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/3366262871 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d41022c0 0x7fa8d4102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/3366262871 >> 172.21.15.105:0/3366262871 conn(0x7fa8d40f7e50 msgr2=0x7fa8d40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/3366262871 shutdown_connections 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/3366262871 wait complete. 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.286+0000 7fa8dc7dc640 1 Processor -- start 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8dc7dc640 1 -- start start 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8dc7dc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 0x7fa8d406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:14.309 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8dc7dc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4102c60 0x7fa8d406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:14.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8dc7dc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d40701f0 con 0x7fa8d41022c0 2024-06-15T00:53:14.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8dc7dc640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d4070360 con 0x7fa8d4102c60 2024-06-15T00:53:14.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 0x7fa8d406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:14.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 0x7fa8d406f770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47320/0 (socket says 172.21.15.105:47320) 2024-06-15T00:53:14.310 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 -- 172.21.15.105:0/2604277663 learned_addr learned my addr 172.21.15.105:0/2604277663 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 -- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4102c60 msgr2=0x7fa8d406fcb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4102c60 0x7fa8d406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 -- 172.21.15.105:0/2604277663 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c4005040 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8da578640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 0x7fa8d406f770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c810 tx=0x7fa8d00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d000a820 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa8d000ae30 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.287+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d40705f0 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.288+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa8d00115c0 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.288+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d41199a0 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.288+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa8d40fda30 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.289+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa8d0012420 con 0x7fa8d41022c0 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.289+0000 7fa8caffd640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa8c0065430 0x7fa8c00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:14.311 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.289+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa8d00829b0 con 0x7fa8d41022c0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.289+0000 7fa8d9d77640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa8c0065430 0x7fa8c00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.290+0000 7fa8caffd640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] conn(0x7fa8c006b1b0 0x7fa8c006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.290+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 --> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa8c006dc70 con 0x7fa8c006b1b0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.290+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa8d0087220 con 0x7fa8d41022c0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.290+0000 7fa8d9d77640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa8c0065430 0x7fa8c00678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4003080 tx=0x7fa8c4009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.290+0000 7fa8dad79640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] conn(0x7fa8c006b1b0 0x7fa8c006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.290+0000 7fa8dad79640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] conn(0x7fa8c006b1b0 0x7fa8c006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.291+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== osd.5 v2:172.21.15.143:6816/4272939362 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fa8c006dc70 con 0x7fa8c006b1b0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 --> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa8d40f97c0 con 0x7fa8c006b1b0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8caffd640 1 -- 172.21.15.105:0/2604277663 <== osd.5 v2:172.21.15.143:6816/4272939362 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fa8d40f97c0 con 0x7fa8c006b1b0 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] conn(0x7fa8c006b1b0 msgr2=0x7fa8c006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] conn(0x7fa8c006b1b0 0x7fa8c006d5c0 crc :-1 s=READY pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.312 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa8c0065430 msgr2=0x7fa8c00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa8c0065430 0x7fa8c00678e0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 crypto rx=0x7fa8c4003080 tx=0x7fa8c4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 msgr2=0x7fa8d406f770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.304+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 0x7fa8d406f770 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 crypto rx=0x7fa8d000c810 tx=0x7fa8d00043c0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 shutdown_connections 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:6816/4272939362,v1:172.21.15.143:6817/4272939362] conn(0x7fa8c006b1b0 0x7fa8c006d5c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa8c0065430 0x7fa8c00678e0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa8d4102c60 0x7fa8d406fcb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 --2- 172.21.15.105:0/2604277663 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa8d41022c0 0x7fa8d406f770 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 >> 172.21.15.105:0/2604277663 conn(0x7fa8d40f7e50 msgr2=0x7fa8d40f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 shutdown_connections 2024-06-15T00:53:14.313 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:14.305+0000 7fa8dc7dc640 1 -- 172.21.15.105:0/2604277663 wait complete. 2024-06-15T00:53:14.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:14 smithi105 ceph-mon[28877]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:14.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:14 smithi143 ceph-mon[35860]: pgmap v143: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:14.996 INFO:teuthology.orchestra.run.smithi105.stdout:171798691848 2024-06-15T00:53:14.996 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.5 2024-06-15T00:53:15.445 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:15.445 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:15.446 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:15.464 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:16.409 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:16 smithi105 ceph-mon[28877]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:16.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:16 smithi143 ceph-mon[35860]: pgmap v144: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.187+0000 7fc158b6f640 1 -- 172.21.15.105:0/2451456480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1541022a0 msgr2=0x7fc154102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.187+0000 7fc158b6f640 1 --2- 172.21.15.105:0/2451456480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1541022a0 0x7fc154102670 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 crypto rx=0x7fc1440048d0 tx=0x7fc144009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.187+0000 7fc158b6f640 1 -- 172.21.15.105:0/2451456480 shutdown_connections 2024-06-15T00:53:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.187+0000 7fc158b6f640 1 --2- 172.21.15.105:0/2451456480 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc154102c40 0x7fc1540fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.187+0000 7fc158b6f640 1 --2- 172.21.15.105:0/2451456480 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc1541022a0 0x7fc154102670 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.201 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.187+0000 7fc158b6f640 1 -- 172.21.15.105:0/2451456480 >> 172.21.15.105:0/2451456480 conn(0x7fc1540f7e50 msgr2=0x7fc1540fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 -- 172.21.15.105:0/2451456480 shutdown_connections 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 -- 172.21.15.105:0/2451456480 wait complete. 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 Processor -- start 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 -- start start 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1541022a0 0x7fc15406f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 0x7fc15406fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc1540701f0 con 0x7fc154102c40 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc158b6f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc154070360 con 0x7fc1541022a0 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc14bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 0x7fc15406fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc14bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 0x7fc15406fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47326/0 (socket says 172.21.15.105:47326) 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.188+0000 7fc14bfff640 1 -- 172.21.15.105:0/227297157 learned_addr learned my addr 172.21.15.105:0/227297157 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:18.202 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc15259c640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1541022a0 0x7fc15406f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14bfff640 1 -- 172.21.15.105:0/227297157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1541022a0 msgr2=0x7fc15406f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14bfff640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1541022a0 0x7fc15406f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14bfff640 1 -- 172.21.15.105:0/227297157 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc144005040 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14bfff640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 0x7fc15406fcb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc13c002a50 tx=0x7fc13c004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13c007c30 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc1540705f0 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc13c00f040 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc13c00da10 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.189+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc154119610 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.190+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fc1540f9890 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.190+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc13c00dc40 con 0x7fc154102c40 2024-06-15T00:53:18.203 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc14affd640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc140065430 0x7fc1400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc13c082220 con 0x7fc154102c40 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc15259c640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc140065430 0x7fc1400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc14affd640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] conn(0x7fc14006b1b0 0x7fc14006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 --> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc14006dc70 con 0x7fc14006b1b0 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fc13c084870 con 0x7fc154102c40 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc152d9d640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] conn(0x7fc14006b1b0 0x7fc14006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.191+0000 7fc15259c640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc140065430 0x7fc1400678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc144007ac0 tx=0x7fc144035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.192+0000 7fc152d9d640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] conn(0x7fc14006b1b0 0x7fc14006d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:18.204 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.192+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== osd.2 v2:172.21.15.105:6818/3804554438 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fc14006dc70 con 0x7fc14006b1b0 2024-06-15T00:53:18.205 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.205+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 --> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fc1540fda10 con 0x7fc14006b1b0 2024-06-15T00:53:18.206 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.205+0000 7fc14affd640 1 -- 172.21.15.105:0/227297157 <== osd.2 v2:172.21.15.105:6818/3804554438 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fc1540fda10 con 0x7fc14006b1b0 2024-06-15T00:53:18.206 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] conn(0x7fc14006b1b0 msgr2=0x7fc14006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] conn(0x7fc14006b1b0 0x7fc14006d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc140065430 msgr2=0x7fc1400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc140065430 0x7fc1400678e0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 crypto rx=0x7fc144007ac0 tx=0x7fc144035040 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 msgr2=0x7fc15406fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 0x7fc15406fcb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 crypto rx=0x7fc13c002a50 tx=0x7fc13c004600 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 shutdown_connections 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6818/3804554438,v1:172.21.15.105:6819/3804554438] conn(0x7fc14006b1b0 0x7fc14006d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc140065430 0x7fc1400678e0 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc154102c40 0x7fc15406fcb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 --2- 172.21.15.105:0/227297157 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc1541022a0 0x7fc15406f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 >> 172.21.15.105:0/227297157 conn(0x7fc1540f7e50 msgr2=0x7fc1540f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 shutdown_connections 2024-06-15T00:53:18.207 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:18.206+0000 7fc158b6f640 1 -- 172.21.15.105:0/227297157 wait complete. 2024-06-15T00:53:18.623 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:18 smithi105 ceph-mon[28877]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:18.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:18 smithi143 ceph-mon[35860]: pgmap v145: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:18.939 INFO:teuthology.orchestra.run.smithi105.stdout:81604378648 2024-06-15T00:53:18.939 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.2 2024-06-15T00:53:20.590 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:20 smithi105 ceph-mon[28877]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:20.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:20 smithi143 ceph-mon[35860]: pgmap v146: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:22.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 -- 172.21.15.105:0/80181485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 msgr2=0x7f8cc4065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/80181485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4065470 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00048d0 tx=0x7f8cc0009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 -- 172.21.15.105:0/80181485 shutdown_connections 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/80181485 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cc40fdeb0 0x7f8cc40fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/80181485 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4065470 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 -- 172.21.15.105:0/80181485 >> 172.21.15.105:0/80181485 conn(0x7f8cc4072110 msgr2=0x7f8cc4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 -- 172.21.15.105:0/80181485 shutdown_connections 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 -- 172.21.15.105:0/80181485 wait complete. 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 Processor -- start 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.367+0000 7f8ccbe11640 1 -- start start 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8ccbe11640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4111eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8ccbe11640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cc40fdeb0 0x7f8cc41123f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8ccbe11640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc4112a40 con 0x7f8cc40650a0 2024-06-15T00:53:22.387 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8ccbe11640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc410cfe0 con 0x7f8cc40fdeb0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4111eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4111eb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47342/0 (socket says 172.21.15.105:47342) 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 -- 172.21.15.105:0/2114481201 learned_addr learned my addr 172.21.15.105:0/2114481201 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 -- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cc40fdeb0 msgr2=0x7f8cc41123f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cc40fdeb0 0x7f8cc41123f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 -- 172.21.15.105:0/2114481201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cc0005040 con 0x7f8cc40650a0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cc9bad640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4111eb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00048a0 tx=0x7f8cc0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cc0038070 con 0x7f8cc40650a0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cc0032d30 con 0x7f8cc40650a0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc410d260 con 0x7f8cc40650a0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cc0009680 con 0x7f8cc40650a0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.368+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc410d720 con 0x7f8cc40650a0 2024-06-15T00:53:22.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.369+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f8cc40fda50 con 0x7f8cc40650a0 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8cc0030020 con 0x7f8cc40650a0 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cba7fc640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cb0065430 0x7f8cb00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8cc00a21f0 con 0x7f8cc40650a0 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cc93ac640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cb0065430 0x7f8cb00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cba7fc640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] conn(0x7f8cb006b1b0 0x7f8cb006d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 --> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f8cb006dc70 con 0x7f8cb006b1b0 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.370+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8cc00a4840 con 0x7f8cc40650a0 2024-06-15T00:53:22.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.371+0000 7f8cc93ac640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cb0065430 0x7f8cb00678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40f8290 tx=0x7f8cb4007cb0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:22.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.371+0000 7f8cca3ae640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] conn(0x7f8cb006b1b0 0x7f8cb006d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.371+0000 7f8cca3ae640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] conn(0x7f8cb006b1b0 0x7f8cb006d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:22.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.372+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== osd.3 v2:172.21.15.143:6800/2670827997 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f8cb006dc70 con 0x7f8cb006b1b0 2024-06-15T00:53:22.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.385+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 --> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f8cc410dc50 con 0x7f8cb006b1b0 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.385+0000 7f8cba7fc640 1 -- 172.21.15.105:0/2114481201 <== osd.3 v2:172.21.15.143:6800/2670827997 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7f8cc410dc50 con 0x7f8cb006b1b0 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] conn(0x7f8cb006b1b0 msgr2=0x7f8cb006d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] conn(0x7f8cb006b1b0 0x7f8cb006d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cb0065430 msgr2=0x7f8cb00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cb0065430 0x7f8cb00678e0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 crypto rx=0x7f8cc40f8290 tx=0x7f8cb4007cb0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 msgr2=0x7f8cc4111eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4111eb0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 crypto rx=0x7f8cc00048a0 tx=0x7f8cc0009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 shutdown_connections 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cb0065430 0x7f8cb00678e0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:6800/2670827997,v1:172.21.15.143:6801/2670827997] conn(0x7f8cb006b1b0 0x7f8cb006d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8cc40fdeb0 0x7f8cc41123f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 --2- 172.21.15.105:0/2114481201 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8cc40650a0 0x7f8cc4111eb0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 >> 172.21.15.105:0/2114481201 conn(0x7f8cc4072110 msgr2=0x7f8cc40f86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:22.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 shutdown_connections 2024-06-15T00:53:22.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.386+0000 7f8ccbe11640 1 -- 172.21.15.105:0/2114481201 wait complete. 2024-06-15T00:53:22.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:22 smithi105 ceph-mon[28877]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:22.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:22 smithi143 ceph-mon[35860]: pgmap v147: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:22.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.790+0000 7fd4aebf4640 1 -- 172.21.15.105:0/1476977318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 msgr2=0x7fd4a80fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.790+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/1476977318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a80fcd80 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 crypto rx=0x7fd4940048d0 tx=0x7fd494009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- 172.21.15.105:0/1476977318 shutdown_connections 2024-06-15T00:53:22.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/1476977318 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a80fcd80 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/1476977318 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4a80fb570 0x7fd4a80fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- 172.21.15.105:0/1476977318 >> 172.21.15.105:0/1476977318 conn(0x7fd4a8072110 msgr2=0x7fd4a8072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- 172.21.15.105:0/1476977318 shutdown_connections 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- 172.21.15.105:0/1476977318 wait complete. 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 Processor -- start 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- start start 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4a80fb570 0x7fd4a8112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a81134b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a8113b40 con 0x7fd4a80fbe80 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.791+0000 7fd4aebf4640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4a810cf40 con 0x7fd4a80fb570 2024-06-15T00:53:22.814 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a81134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a81134b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47352/0 (socket says 172.21.15.105:47352) 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 -- 172.21.15.105:0/3572707424 learned_addr learned my addr 172.21.15.105:0/3572707424 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4ac990640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4a80fb570 0x7fd4a8112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 -- 172.21.15.105:0/3572707424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4a80fb570 msgr2=0x7fd4a8112f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4a80fb570 0x7fd4a8112f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 -- 172.21.15.105:0/3572707424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd494005040 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a7fff640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a81134b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd4940048a0 tx=0x7fd494009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd494038070 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd494032d30 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4a810d1c0 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd494009680 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.792+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4a810d680 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.793+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fd4a810dc20 con 0x7fd4a80fbe80 2024-06-15T00:53:22.815 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd494030020 con 0x7fd4a80fbe80 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4a57fa640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd498065430 0x7fd4980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd4940a2300 con 0x7fd4a80fbe80 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4ac990640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd498065430 0x7fd4980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4a57fa640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] conn(0x7fd49806b1b0 0x7fd49806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 --> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd49806dc70 con 0x7fd49806b1b0 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.794+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fd4940a48f0 con 0x7fd4a80fbe80 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.795+0000 7fd4ad191640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] conn(0x7fd49806b1b0 0x7fd49806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.795+0000 7fd4ac990640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd498065430 0x7fd4980678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006fd0 tx=0x7fd49c008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.795+0000 7fd4ad191640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] conn(0x7fd49806b1b0 0x7fd49806d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.795+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== osd.1 v2:172.21.15.105:6810/3566937114 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fd49806dc70 con 0x7fd49806b1b0 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.808+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 --> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fd4a810dde0 con 0x7fd49806b1b0 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4a57fa640 1 -- 172.21.15.105:0/3572707424 <== osd.1 v2:172.21.15.105:6810/3566937114 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fd4a810dde0 con 0x7fd49806b1b0 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] conn(0x7fd49806b1b0 msgr2=0x7fd49806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] conn(0x7fd49806b1b0 0x7fd49806d5c0 crc :-1 s=READY pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd498065430 msgr2=0x7fd4980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd498065430 0x7fd4980678e0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 crypto rx=0x7fd49c006fd0 tx=0x7fd49c008040 comp rx=0 tx=0).stop 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 msgr2=0x7fd4a81134b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a81134b0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 crypto rx=0x7fd4940048a0 tx=0x7fd494009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 shutdown_connections 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd498065430 0x7fd4980678e0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:6810/3566937114,v1:172.21.15.105:6811/3566937114] conn(0x7fd49806b1b0 0x7fd49806d5c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.817 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd4a80fbe80 0x7fd4a81134b0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 --2- 172.21.15.105:0/3572707424 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd4a80fb570 0x7fd4a8112f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:22.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 >> 172.21.15.105:0/3572707424 conn(0x7fd4a8072110 msgr2=0x7fd4a80ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:22.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 shutdown_connections 2024-06-15T00:53:22.818 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:22.809+0000 7fd4aebf4640 1 -- 172.21.15.105:0/3572707424 wait complete. 2024-06-15T00:53:23.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.166+0000 7f7d63e99640 1 -- 172.21.15.105:0/3612400101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 msgr2=0x7f7d5c0fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:23.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.166+0000 7f7d63e99640 1 --2- 172.21.15.105:0/3612400101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c0fb940 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 crypto rx=0x7f7d580048d0 tx=0x7f7d58009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.166+0000 7f7d63e99640 1 -- 172.21.15.105:0/3612400101 shutdown_connections 2024-06-15T00:53:23.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.166+0000 7f7d63e99640 1 --2- 172.21.15.105:0/3612400101 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d5c0fbe80 0x7f7d5c0fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.190 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.166+0000 7f7d63e99640 1 --2- 172.21.15.105:0/3612400101 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c0fb940 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.166+0000 7f7d63e99640 1 -- 172.21.15.105:0/3612400101 >> 172.21.15.105:0/3612400101 conn(0x7f7d5c072110 msgr2=0x7f7d5c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 -- 172.21.15.105:0/3612400101 shutdown_connections 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 -- 172.21.15.105:0/3612400101 wait complete. 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 Processor -- start 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 -- start start 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d5c0fbe80 0x7f7d5c113480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c113a80 con 0x7f7d5c0fb570 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d63e99640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d5c10cec0 con 0x7f7d5c0fbe80 2024-06-15T00:53:23.191 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d61c35640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d61c35640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60756/0 (socket says 172.21.15.105:60756) 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.167+0000 7f7d61c35640 1 -- 172.21.15.105:0/1363654409 learned_addr learned my addr 172.21.15.105:0/1363654409 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d61434640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d5c0fbe80 0x7f7d5c113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d61c35640 1 -- 172.21.15.105:0/1363654409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d5c0fbe80 msgr2=0x7f7d5c113480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d61c35640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d5c0fbe80 0x7f7d5c113480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d61c35640 1 -- 172.21.15.105:0/1363654409 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d58005040 con 0x7f7d5c0fb570 2024-06-15T00:53:23.192 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d61c35640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c112f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d580048a0 tx=0x7f7d58009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d58038070 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d5802cef0 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d5c10d140 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7d58009640 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.168+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d5c10d680 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.169+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7d5c10dbb0 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.169+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7d58030020 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d527fc640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7d48065430 0x7f7d480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7d580a2800 con 0x7f7d5c0fb570 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d61434640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7d48065430 0x7f7d480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d527fc640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] conn(0x7f7d4806b1b0 0x7f7d4806d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:23.193 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 --> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7d4806dc70 con 0x7f7d4806b1b0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7d580a7220 con 0x7f7d5c0fb570 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d62436640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] conn(0x7f7d4806b1b0 0x7f7d4806d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d61434640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7d48065430 0x7f7d480678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c10e800 tx=0x7f7d4c00b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.170+0000 7f7d62436640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] conn(0x7f7d4806b1b0 0x7f7d4806d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.171+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== osd.0 v2:172.21.15.105:6802/9359752 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7f7d4806dc70 con 0x7f7d4806b1b0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.184+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 --> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f7d5c0fc920 con 0x7f7d4806b1b0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.184+0000 7f7d527fc640 1 -- 172.21.15.105:0/1363654409 <== osd.0 v2:172.21.15.105:6802/9359752 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f7d5c0fc920 con 0x7f7d4806b1b0 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.184+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] conn(0x7f7d4806b1b0 msgr2=0x7f7d4806d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.184+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] conn(0x7f7d4806b1b0 0x7f7d4806d5c0 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7d48065430 msgr2=0x7f7d480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7d48065430 0x7f7d480678e0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 crypto rx=0x7f7d5c10e800 tx=0x7f7d4c00b040 comp rx=0 tx=0).stop 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 msgr2=0x7f7d5c112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:23.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c112f40 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 crypto rx=0x7f7d580048a0 tx=0x7f7d58009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 shutdown_connections 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6802/9359752,v1:172.21.15.105:6803/9359752] conn(0x7f7d4806b1b0 0x7f7d4806d5c0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7d48065430 0x7f7d480678e0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7d5c0fbe80 0x7f7d5c113480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 --2- 172.21.15.105:0/1363654409 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7d5c0fb570 0x7f7d5c112f40 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 >> 172.21.15.105:0/1363654409 conn(0x7f7d5c072110 msgr2=0x7f7d5c0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 shutdown_connections 2024-06-15T00:53:23.195 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:23.185+0000 7f7d63e99640 1 -- 172.21.15.105:0/1363654409 wait complete. 2024-06-15T00:53:23.963 INFO:teuthology.orchestra.run.smithi105.stdout:38654705699 2024-06-15T00:53:23.964 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.0 2024-06-15T00:53:24.210 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:24.210 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:24.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:24 smithi105 ceph-mon[28877]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:24.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:24 smithi143 ceph-mon[35860]: pgmap v148: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:24.648 INFO:teuthology.orchestra.run.smithi105.stdout:60129542173 2024-06-15T00:53:24.649 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.1 2024-06-15T00:53:25.062 INFO:teuthology.orchestra.run.smithi105.stdout:115964117011 2024-06-15T00:53:25.062 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.3 2024-06-15T00:53:25.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2732288348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 msgr2=0x7fcb68064d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:25.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2732288348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68064d40 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 crypto rx=0x7fcb640048d0 tx=0x7fcb64009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2732288348 shutdown_connections 2024-06-15T00:53:25.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2732288348 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68065280 0x7fcb680656d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2732288348 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68064d40 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.397 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2732288348 >> 172.21.15.105:0/2732288348 conn(0x7fcb68072110 msgr2=0x7fcb68072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2732288348 shutdown_connections 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.261+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2732288348 wait complete. 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6f98c640 1 Processor -- start 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6f98c640 1 -- start start 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6f98c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68065280 0x7fcb681194f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6f98c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68119a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6f98c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb6811a080 con 0x7fcb681012a0 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6f98c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb68113490 con 0x7fcb68065280 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68119a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68119a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60784/0 (socket says 172.21.15.105:60784) 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 -- 172.21.15.105:0/2595484228 learned_addr learned my addr 172.21.15.105:0/2595484228 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:25.398 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 -- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68065280 msgr2=0x7fcb681194f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68065280 0x7fcb681194f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 -- 172.21.15.105:0/2595484228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb64005040 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.262+0000 7fcb6cf27640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68119a30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c810 tx=0x7fcb580043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.263+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb5800a820 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.263+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb5800ae30 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.263+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb68113770 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.263+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcb580115c0 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.263+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb68113d10 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.264+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fcb68114290 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.264+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fcb58012420 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.264+0000 7fcb5dffb640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcb54065430 0x7fcb540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.264+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcb58082ab0 con 0x7fcb681012a0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.264+0000 7fcb6d728640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcb54065430 0x7fcb540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:25.399 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.265+0000 7fcb5dffb640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] conn(0x7fcb5406b1b0 0x7fcb5406d5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.265+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 --> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fcb5406dc70 con 0x7fcb5406b1b0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.265+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_get_version_reply(handle=1 version=41) v2 ==== 24+0+0 (secure 0 0 0) 0x7fcb58087220 con 0x7fcb681012a0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.265+0000 7fcb6d728640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcb54065430 0x7fcb540678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb64003080 tx=0x7fcb64009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.265+0000 7fcb6df29640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] conn(0x7fcb5406b1b0 0x7fcb5406d5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.265+0000 7fcb6df29640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] conn(0x7fcb5406b1b0 0x7fcb5406d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.266+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== osd.4 v2:172.21.15.143:6808/3256896185 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+24147 (crc 0 0 0) 0x7fcb5406dc70 con 0x7fcb5406b1b0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.279+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 --> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fcb680fabb0 con 0x7fcb5406b1b0 2024-06-15T00:53:25.400 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.279+0000 7fcb5dffb640 1 -- 172.21.15.105:0/2595484228 <== osd.4 v2:172.21.15.143:6808/3256896185 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+12 (crc 0 0 0) 0x7fcb680fabb0 con 0x7fcb5406b1b0 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] conn(0x7fcb5406b1b0 msgr2=0x7fcb5406d5c0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] conn(0x7fcb5406b1b0 0x7fcb5406d5c0 crc :-1 s=READY pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcb54065430 msgr2=0x7fcb540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcb54065430 0x7fcb540678e0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 crypto rx=0x7fcb64003080 tx=0x7fcb64009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 msgr2=0x7fcb68119a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68119a30 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 crypto rx=0x7fcb5800c810 tx=0x7fcb580043c0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 shutdown_connections 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcb54065430 0x7fcb540678e0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:6808/3256896185,v1:172.21.15.143:6809/3256896185] conn(0x7fcb5406b1b0 0x7fcb5406d5c0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcb681012a0 0x7fcb68119a30 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 --2- 172.21.15.105:0/2595484228 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcb68065280 0x7fcb681194f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 >> 172.21.15.105:0/2595484228 conn(0x7fcb68072110 msgr2=0x7fcb680fa1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 shutdown_connections 2024-06-15T00:53:25.401 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:25.280+0000 7fcb6f98c640 1 -- 172.21.15.105:0/2595484228 wait complete. 2024-06-15T00:53:25.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:25.715 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:26.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:26 smithi143 ceph-mon[35860]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:26.718 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:26.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:26 smithi105 ceph-mon[28877]: pgmap v149: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:28.011 INFO:teuthology.orchestra.run.smithi105.stdout:150323855375 2024-06-15T00:53:28.012 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph osd last-stat-seq osd.4 2024-06-15T00:53:28.483 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:28 smithi105 ceph-mon[28877]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:28.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:28 smithi143 ceph-mon[35860]: pgmap v150: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:29.405 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:29.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 -- 172.21.15.105:0/2965269310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e41011a0 msgr2=0x7f26e4101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 --2- 172.21.15.105:0/2965269310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e41011a0 0x7f26e4101570 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 crypto rx=0x7f26d40048d0 tx=0x7f26d4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 -- 172.21.15.105:0/2965269310 shutdown_connections 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 --2- 172.21.15.105:0/2965269310 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e4101b40 0x7f26e40fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 --2- 172.21.15.105:0/2965269310 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e41011a0 0x7f26e4101570 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 -- 172.21.15.105:0/2965269310 >> 172.21.15.105:0/2965269310 conn(0x7f26e40f6d50 msgr2=0x7f26e40f9160 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 -- 172.21.15.105:0/2965269310 shutdown_connections 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.478+0000 7f26e8992640 1 -- 172.21.15.105:0/2965269310 wait complete. 2024-06-15T00:53:29.490 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e8992640 1 Processor -- start 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e8992640 1 -- start start 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e8992640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e41011a0 0x7f26e410bd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e8992640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 0x7f26e410c2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e8992640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e410c7f0 con 0x7f26e4101b40 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e8992640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f26e410c930 con 0x7f26e41011a0 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 0x7f26e410c2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 0x7f26e410c2b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60800/0 (socket says 172.21.15.105:60800) 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 -- 172.21.15.105:0/168754341 learned_addr learned my addr 172.21.15.105:0/168754341 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:29.491 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 -- 172.21.15.105:0/168754341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e41011a0 msgr2=0x7f26e410bd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e41011a0 0x7f26e410bd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 -- 172.21.15.105:0/168754341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26d4005040 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26e1d9b640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 0x7f26e410c2b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b790 tx=0x7f26d80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.479+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d800c700 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.480+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f26d800cd10 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.480+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f26e41193e0 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.480+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f26d8013370 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.480+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26e4119760 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.480+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26e40fc910 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.481+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f26d8012070 con 0x7f26e4101b40 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.481+0000 7f26d2ffd640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f26cc065430 0x7f26cc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:29.492 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.481+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f26d8082100 con 0x7f26e4101b40 2024-06-15T00:53:29.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.481+0000 7f26e259c640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f26cc065430 0x7f26cc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:29.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.482+0000 7f26e259c640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f26cc065430 0x7f26cc0678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26d402a530 tx=0x7f26d4009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:29.493 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.485+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f26d80505b0 con 0x7f26e4101b40 2024-06-15T00:53:29.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.655+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f26e40fa990 con 0x7f26e4101b40 2024-06-15T00:53:29.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.656+0000 7f26d2ffd640 1 -- 172.21.15.105:0/168754341 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f26d8053bd0 con 0x7f26e4101b40 2024-06-15T00:53:29.656 INFO:teuthology.orchestra.run.smithi105.stdout:81604378650 2024-06-15T00:53:29.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f26cc065430 msgr2=0x7f26cc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:29.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f26cc065430 0x7f26cc0678e0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 crypto rx=0x7f26d402a530 tx=0x7f26d4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 msgr2=0x7f26e410c2b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:29.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 0x7f26e410c2b0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 crypto rx=0x7f26d800b790 tx=0x7f26d80091d0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.658 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 shutdown_connections 2024-06-15T00:53:29.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f26cc065430 0x7f26cc0678e0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26e4101b40 0x7f26e410c2b0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 --2- 172.21.15.105:0/168754341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26e41011a0 0x7f26e410bd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:29.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 >> 172.21.15.105:0/168754341 conn(0x7f26e40f6d50 msgr2=0x7f26e40f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:29.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 shutdown_connections 2024-06-15T00:53:29.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:29.658+0000 7f26e8992640 1 -- 172.21.15.105:0/168754341 wait complete. 2024-06-15T00:53:30.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:30 smithi143 ceph-mon[35860]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:30.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:30 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/168754341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-15T00:53:30.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:30 smithi105 ceph-mon[28877]: pgmap v151: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:30.673 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:30 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/168754341' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 2}]: dispatch 2024-06-15T00:53:30.789 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 -- 172.21.15.105:0/926122588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c102c70 msgr2=0x7fda1c0fc300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 --2- 172.21.15.105:0/926122588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c102c70 0x7fda1c0fc300 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 crypto rx=0x7fda100048d0 tx=0x7fda10009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 -- 172.21.15.105:0/926122588 shutdown_connections 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 --2- 172.21.15.105:0/926122588 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c102c70 0x7fda1c0fc300 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 --2- 172.21.15.105:0/926122588 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c1022d0 0x7fda1c1026a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 -- 172.21.15.105:0/926122588 >> 172.21.15.105:0/926122588 conn(0x7fda1c0f7e80 msgr2=0x7fda1c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 -- 172.21.15.105:0/926122588 shutdown_connections 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.757+0000 7fda227eb640 1 -- 172.21.15.105:0/926122588 wait complete. 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda227eb640 1 Processor -- start 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda227eb640 1 -- start start 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda227eb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 0x7fda1c112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda227eb640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c102c70 0x7fda1c113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:30.790 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda227eb640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c113b90 con 0x7fda1c1022d0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda227eb640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda1c10cf40 con 0x7fda1c102c70 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 0x7fda1c112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda1bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 0x7fda1c112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60820/0 (socket says 172.21.15.105:60820) 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.758+0000 7fda1bfff640 1 -- 172.21.15.105:0/4169951088 learned_addr learned my addr 172.21.15.105:0/4169951088 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda1b7fe640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c102c70 0x7fda1c113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda1bfff640 1 -- 172.21.15.105:0/4169951088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c102c70 msgr2=0x7fda1c113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda1bfff640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c102c70 0x7fda1c113500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda1bfff640 1 -- 172.21.15.105:0/4169951088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda10005040 con 0x7fda1c1022d0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda1bfff640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 0x7fda1c112fc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00b790 tx=0x7fda0c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda0c00c700 con 0x7fda1c1022d0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda1c10d220 con 0x7fda1c1022d0 2024-06-15T00:53:30.791 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda0c00cd10 con 0x7fda1c1022d0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fda0c013370 con 0x7fda1c1022d0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.759+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda1c10d740 con 0x7fda1c1022d0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.760+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda1c0fda40 con 0x7fda1c1022d0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.761+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fda0c012070 con 0x7fda1c1022d0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.761+0000 7fda18ff9640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fda04065430 0x7fda040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.761+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fda0c082170 con 0x7fda1c1022d0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.761+0000 7fda1b7fe640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fda04065430 0x7fda040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.762+0000 7fda1b7fe640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fda04065430 0x7fda040678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda1000b020 tx=0x7fda10035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:30.792 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.765+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fda0c050620 con 0x7fda1c1022d0 2024-06-15T00:53:30.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.936+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fda1c04b4b0 con 0x7fda1c1022d0 2024-06-15T00:53:30.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.936+0000 7fda18ff9640 1 -- 172.21.15.105:0/4169951088 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fda0c053c40 con 0x7fda1c1022d0 2024-06-15T00:53:30.936 INFO:teuthology.orchestra.run.smithi105.stdout:171798691851 2024-06-15T00:53:30.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fda04065430 msgr2=0x7fda040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fda04065430 0x7fda040678e0 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 crypto rx=0x7fda1000b020 tx=0x7fda10035040 comp rx=0 tx=0).stop 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 msgr2=0x7fda1c112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 0x7fda1c112fc0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 crypto rx=0x7fda0c00b790 tx=0x7fda0c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 shutdown_connections 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fda04065430 0x7fda040678e0 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fda1c102c70 0x7fda1c113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 --2- 172.21.15.105:0/4169951088 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fda1c1022d0 0x7fda1c112fc0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 >> 172.21.15.105:0/4169951088 conn(0x7fda1c0f7e80 msgr2=0x7fda1c0f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 shutdown_connections 2024-06-15T00:53:30.939 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:30.938+0000 7fda227eb640 1 -- 172.21.15.105:0/4169951088 wait complete. 2024-06-15T00:53:31.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:31 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/4169951088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-15T00:53:31.681 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:31 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/4169951088' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 5}]: dispatch 2024-06-15T00:53:31.970 INFO:tasks.cephadm.ceph_manager.ceph:need seq 171798691848 got 171798691851 for osd.5 2024-06-15T00:53:31.970 DEBUG:teuthology.parallel:result is None 2024-06-15T00:53:32.188 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:32.188 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:32.354 INFO:tasks.cephadm.ceph_manager.ceph:need seq 81604378648 got 81604378650 for osd.2 2024-06-15T00:53:32.354 DEBUG:teuthology.parallel:result is None 2024-06-15T00:53:32.470 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:32 smithi105 ceph-mon[28877]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:32.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:32 smithi143 ceph-mon[35860]: pgmap v152: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:33.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:33 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2433880465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-15T00:53:33.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:33 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2433880465' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 0}]: dispatch 2024-06-15T00:53:34.381 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 -- 172.21.15.105:0/575548885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fb0c0 msgr2=0x7f0c440fb490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/575548885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fb0c0 0x7f0c440fb490 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 crypto rx=0x7f0c400048d0 tx=0x7f0c40009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 -- 172.21.15.105:0/575548885 shutdown_connections 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/575548885 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440fba60 0x7f0c44100080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/575548885 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fb0c0 0x7f0c440fb490 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 -- 172.21.15.105:0/575548885 >> 172.21.15.105:0/575548885 conn(0x7f0c440f6d60 msgr2=0x7f0c440f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 -- 172.21.15.105:0/575548885 shutdown_connections 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.734+0000 7f0c4c31e640 1 -- 172.21.15.105:0/575548885 wait complete. 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.735+0000 7f0c4c31e640 1 Processor -- start 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.735+0000 7f0c4c31e640 1 -- start start 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.735+0000 7f0c4c31e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440fb0c0 0x7f0c441195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.735+0000 7f0c4c31e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 0x7f0c44119b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:34.382 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.735+0000 7f0c4c31e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c4411a180 con 0x7f0c440fba60 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.735+0000 7f0c4c31e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c44113570 con 0x7f0c440fb0c0 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 0x7f0c44119b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 0x7f0c44119b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:60838/0 (socket says 172.21.15.105:60838) 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 -- 172.21.15.105:0/2433880465 learned_addr learned my addr 172.21.15.105:0/2433880465 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c4a0ba640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440fb0c0 0x7f0c441195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 -- 172.21.15.105:0/2433880465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440fb0c0 msgr2=0x7f0c441195d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440fb0c0 0x7f0c441195d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 -- 172.21.15.105:0/2433880465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40005040 con 0x7f0c440fba60 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c498b9640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 0x7f0c44119b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0c34002a80 tx=0x7f0c34004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c34007c30 con 0x7f0c440fba60 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0c3400f040 con 0x7f0c440fba60 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c44113850 con 0x7f0c440fba60 2024-06-15T00:53:34.383 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0c3400db10 con 0x7f0c440fba60 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.736+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c44113d70 con 0x7f0c440fba60 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.737+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c44066090 con 0x7f0c440fba60 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.738+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0c3400dc70 con 0x7f0c440fba60 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.738+0000 7f0c3affd640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0c30065430 0x7f0c300678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.738+0000 7f0c4a0ba640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0c30065430 0x7f0c300678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.738+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0c34081c90 con 0x7f0c440fba60 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stdout:38654705700 2024-06-15T00:53:34.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.739+0000 7f0c4a0ba640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0c30065430 0x7f0c300678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0c4000b020 tx=0x7f0c40035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.742+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0c3404cdf0 con 0x7f0c440fba60 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.911+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f0c440fa9a0 con 0x7f0c440fba60 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.912+0000 7f0c3affd640 1 -- 172.21.15.105:0/2433880465 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f0c34050270 con 0x7f0c440fba60 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0c30065430 msgr2=0x7f0c300678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0c30065430 0x7f0c300678e0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 crypto rx=0x7f0c4000b020 tx=0x7f0c40035040 comp rx=0 tx=0).stop 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 msgr2=0x7f0c44119b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 0x7f0c44119b30 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 crypto rx=0x7f0c34002a80 tx=0x7f0c34004600 comp rx=0 tx=0).stop 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 shutdown_connections 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0c30065430 0x7f0c300678e0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0c440fba60 0x7f0c44119b30 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 --2- 172.21.15.105:0/2433880465 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0c440fb0c0 0x7f0c441195d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 >> 172.21.15.105:0/2433880465 conn(0x7f0c440f6d60 msgr2=0x7f0c440f8380 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 shutdown_connections 2024-06-15T00:53:34.385 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:32.914+0000 7f0c4c31e640 1 -- 172.21.15.105:0/2433880465 wait complete. 2024-06-15T00:53:34.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:34 smithi105 ceph-mon[28877]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:34.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:34 smithi143 ceph-mon[35860]: pgmap v153: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:35.069 INFO:tasks.cephadm.ceph_manager.ceph:need seq 38654705699 got 38654705700 for osd.0 2024-06-15T00:53:35.069 DEBUG:teuthology.parallel:result is None 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 -- 172.21.15.105:0/333090334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0650a0 msgr2=0x7f29ec065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 --2- 172.21.15.105:0/333090334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0650a0 0x7f29ec065470 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 crypto rx=0x7f29e80048d0 tx=0x7f29e8009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 -- 172.21.15.105:0/333090334 shutdown_connections 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 --2- 172.21.15.105:0/333090334 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec0fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 --2- 172.21.15.105:0/333090334 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0650a0 0x7f29ec065470 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 -- 172.21.15.105:0/333090334 >> 172.21.15.105:0/333090334 conn(0x7f29ec072110 msgr2=0x7f29ec072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 -- 172.21.15.105:0/333090334 shutdown_connections 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 -- 172.21.15.105:0/333090334 wait complete. 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.508+0000 7f29f373a640 1 Processor -- start 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f373a640 1 -- start start 2024-06-15T00:53:35.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f373a640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec0650a0 0x7f29ec112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f373a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec113440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f373a640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec113a40 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f373a640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29ec10ce80 con 0x7f29ec0650a0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec113440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec113440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43090/0 (socket says 172.21.15.105:43090) 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 -- 172.21.15.105:0/3502353830 learned_addr learned my addr 172.21.15.105:0/3502353830 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f14d6640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec0650a0 0x7f29ec112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 -- 172.21.15.105:0/3502353830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec0650a0 msgr2=0x7f29ec112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec0650a0 0x7f29ec112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 -- 172.21.15.105:0/3502353830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29e8005040 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.509+0000 7f29f0cd5640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec113440 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00c830 tx=0x7f29dc004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.510+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29dc00a850 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.515 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.510+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29dc00ae60 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.510+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29ec10d160 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.510+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f29dc0113a0 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.510+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29ec10d700 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.511+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29ec0fda50 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.511+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f29dc012400 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.512+0000 7f29e1ffb640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f29d8065430 0x7f29d80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.512+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f29dc0826e0 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.512+0000 7f29f14d6640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f29d8065430 0x7f29d80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:35.516 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.512+0000 7f29f14d6640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f29d8065430 0x7f29d80678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f29e8004540 tx=0x7f29e8035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:35.517 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.516+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f29dc050cd0 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.685 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.685+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7f29ec072c20 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.686 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.685+0000 7f29e1ffb640 1 -- 172.21.15.105:0/3502353830 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7f29dc0542f0 con 0x7f29ec0fdeb0 2024-06-15T00:53:35.686 INFO:teuthology.orchestra.run.smithi105.stdout:150323855377 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f29d8065430 msgr2=0x7f29d80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f29d8065430 0x7f29d80678e0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 crypto rx=0x7f29e8004540 tx=0x7f29e8035040 comp rx=0 tx=0).stop 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 msgr2=0x7f29ec113440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec113440 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 crypto rx=0x7f29dc00c830 tx=0x7f29dc004430 comp rx=0 tx=0).stop 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 shutdown_connections 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f29d8065430 0x7f29d80678e0 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f29ec0fdeb0 0x7f29ec113440 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 --2- 172.21.15.105:0/3502353830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f29ec0650a0 0x7f29ec112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 >> 172.21.15.105:0/3502353830 conn(0x7f29ec072110 msgr2=0x7f29ec0f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 shutdown_connections 2024-06-15T00:53:35.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:35.687+0000 7f29f373a640 1 -- 172.21.15.105:0/3502353830 wait complete. 2024-06-15T00:53:36.629 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:36 smithi105 ceph-mon[28877]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:36.629 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:36 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3502353830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-15T00:53:36.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:36 smithi143 ceph-mon[35860]: pgmap v154: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:36.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:36 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3502353830' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 4}]: dispatch 2024-06-15T00:53:36.653 INFO:tasks.cephadm.ceph_manager.ceph:need seq 150323855375 got 150323855377 for osd.4 2024-06-15T00:53:36.653 DEBUG:teuthology.parallel:result is None 2024-06-15T00:53:37.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.739+0000 7fe9ec384640 1 -- 172.21.15.105:0/456716521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 msgr2=0x7fe9e40fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:37.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.739+0000 7fe9ec384640 1 --2- 172.21.15.105:0/456716521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e40fcd80 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40048d0 tx=0x7fe9d4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- 172.21.15.105:0/456716521 shutdown_connections 2024-06-15T00:53:37.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 --2- 172.21.15.105:0/456716521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e40fcd80 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 --2- 172.21.15.105:0/456716521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e40fb570 0x7fe9e40fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.754 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- 172.21.15.105:0/456716521 >> 172.21.15.105:0/456716521 conn(0x7fe9e4072110 msgr2=0x7fe9e4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- 172.21.15.105:0/456716521 shutdown_connections 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- 172.21.15.105:0/456716521 wait complete. 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 Processor -- start 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- start start 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e40fb570 0x7fe9e4112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e4113480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e4113a80 con 0x7fe9e40fbe80 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.740+0000 7fe9ec384640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9e410cec0 con 0x7fe9e40fb570 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e4113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:37.755 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e4113480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43100/0 (socket says 172.21.15.105:43100) 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 -- 172.21.15.105:0/3153295682 learned_addr learned my addr 172.21.15.105:0/3153295682 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9ea120640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e40fb570 0x7fe9e4112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 -- 172.21.15.105:0/3153295682 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e40fb570 msgr2=0x7fe9e4112f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e40fb570 0x7fe9e4112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 -- 172.21.15.105:0/3153295682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9d4005040 con 0x7fe9e40fbe80 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9e991f640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e4113480 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40048a0 tx=0x7fe9d4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d4038070 con 0x7fe9e40fbe80 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9e410d140 con 0x7fe9e40fbe80 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9d402cef0 con 0x7fe9e40fbe80 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fe9d4009640 con 0x7fe9e40fbe80 2024-06-15T00:53:37.756 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.741+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9e410d680 con 0x7fe9e40fbe80 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.742+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9e40fc920 con 0x7fe9e40fbe80 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.742+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fe9d4030020 con 0x7fe9e40fbe80 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.743+0000 7fe9daffd640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fe9d0065430 0x7fe9d00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.743+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fe9d40a2930 con 0x7fe9e40fbe80 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.743+0000 7fe9ea120640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fe9d0065430 0x7fe9d00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.743+0000 7fe9ea120640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fe9d0065430 0x7fe9d00678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0004ce0 tx=0x7fe9e000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:37.757 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.747+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fe9d4070de0 con 0x7fe9e40fbe80 2024-06-15T00:53:37.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.915+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7fe9e410e5b0 con 0x7fe9e40fbe80 2024-06-15T00:53:37.915 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.915+0000 7fe9daffd640 1 -- 172.21.15.105:0/3153295682 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+13 (secure 0 0 0) 0x7fe9d4074400 con 0x7fe9e40fbe80 2024-06-15T00:53:37.916 INFO:teuthology.orchestra.run.smithi105.stdout:115964117014 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fe9d0065430 msgr2=0x7fe9d00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fe9d0065430 0x7fe9d00678e0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 crypto rx=0x7fe9e0004ce0 tx=0x7fe9e000b040 comp rx=0 tx=0).stop 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 msgr2=0x7fe9e4113480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e4113480 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 crypto rx=0x7fe9d40048a0 tx=0x7fe9d4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 shutdown_connections 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fe9d0065430 0x7fe9d00678e0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.918 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fe9e40fbe80 0x7fe9e4113480 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 --2- 172.21.15.105:0/3153295682 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fe9e40fb570 0x7fe9e4112f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.917+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 >> 172.21.15.105:0/3153295682 conn(0x7fe9e4072110 msgr2=0x7fe9e41000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:37.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.918+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 shutdown_connections 2024-06-15T00:53:37.919 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.918+0000 7fe9ec384640 1 -- 172.21.15.105:0/3153295682 wait complete. 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.954+0000 7f07aac46640 1 -- 172.21.15.105:0/2780590315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 msgr2=0x7f07a4101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.954+0000 7f07aac46640 1 --2- 172.21.15.105:0/2780590315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4101570 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 crypto rx=0x7f07980048d0 tx=0x7f0798009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.954+0000 7f07aac46640 1 -- 172.21.15.105:0/2780590315 shutdown_connections 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.954+0000 7f07aac46640 1 --2- 172.21.15.105:0/2780590315 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a4101b40 0x7f07a40fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.954+0000 7f07aac46640 1 --2- 172.21.15.105:0/2780590315 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4101570 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.954+0000 7f07aac46640 1 -- 172.21.15.105:0/2780590315 >> 172.21.15.105:0/2780590315 conn(0x7f07a40f6d90 msgr2=0x7f07a40f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 -- 172.21.15.105:0/2780590315 shutdown_connections 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 -- 172.21.15.105:0/2780590315 wait complete. 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 Processor -- start 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 -- start start 2024-06-15T00:53:37.997 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4102020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a4101b40 0x7f07a4102560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4102aa0 con 0x7f07a41011a0 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.955+0000 7f07aac46640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a4102c10 con 0x7f07a4101b40 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4102020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4102020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43120/0 (socket says 172.21.15.105:43120) 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 -- 172.21.15.105:0/600198977 learned_addr learned my addr 172.21.15.105:0/600198977 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a3fff640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a4101b40 0x7f07a4102560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 -- 172.21.15.105:0/600198977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a4101b40 msgr2=0x7f07a4102560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a4101b40 0x7f07a4102560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 -- 172.21.15.105:0/600198977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0798005040 con 0x7f07a41011a0 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a89e2640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4102020 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f07980048a0 tx=0x7f0798009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0798038070 con 0x7f07a41011a0 2024-06-15T00:53:37.998 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a4102e40 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f079802cef0 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0798009640 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.956+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a4119540 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.957+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a40fc910 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.957+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0798030020 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.958+0000 7f07a17fa640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0794065430 0x7f07940678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.958+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f07980a2910 con 0x7f07a41011a0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.958+0000 7f07a3fff640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0794065430 0x7f07940678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.959+0000 7f07a3fff640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0794065430 0x7f07940678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07a4103960 tx=0x7f0790009ed0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:37.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:37.962+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0798070dc0 con 0x7f07a41011a0 2024-06-15T00:53:38.131 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.131+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f07a4103710 con 0x7f07a41011a0 2024-06-15T00:53:38.132 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.131+0000 7f07a17fa640 1 -- 172.21.15.105:0/600198977 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f07980743e0 con 0x7f07a41011a0 2024-06-15T00:53:38.132 INFO:teuthology.orchestra.run.smithi105.stdout:60129542176 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0794065430 msgr2=0x7f07940678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0794065430 0x7f07940678e0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 crypto rx=0x7f07a4103960 tx=0x7f0790009ed0 comp rx=0 tx=0).stop 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 msgr2=0x7f07a4102020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4102020 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 crypto rx=0x7f07980048a0 tx=0x7f0798009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 shutdown_connections 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0794065430 0x7f07940678e0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f07a4101b40 0x7f07a4102560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 --2- 172.21.15.105:0/600198977 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f07a41011a0 0x7f07a4102020 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:38.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 >> 172.21.15.105:0/600198977 conn(0x7f07a40f6d90 msgr2=0x7f07a40f84a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:38.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 shutdown_connections 2024-06-15T00:53:38.135 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:38.133+0000 7f07aac46640 1 -- 172.21.15.105:0/600198977 wait complete. 2024-06-15T00:53:38.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:38 smithi105 ceph-mon[28877]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:38.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:38 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3153295682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-15T00:53:38.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:38 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/600198977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-15T00:53:38.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:38 smithi143 ceph-mon[35860]: pgmap v155: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:38.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:38 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3153295682' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 3}]: dispatch 2024-06-15T00:53:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:38 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/600198977' entity='client.admin' cmd=[{"prefix": "osd last-stat-seq", "id": 1}]: dispatch 2024-06-15T00:53:38.659 INFO:tasks.cephadm.ceph_manager.ceph:need seq 115964117011 got 115964117014 for osd.3 2024-06-15T00:53:38.660 DEBUG:teuthology.parallel:result is None 2024-06-15T00:53:39.382 INFO:tasks.cephadm.ceph_manager.ceph:need seq 60129542173 got 60129542176 for osd.1 2024-06-15T00:53:39.382 DEBUG:teuthology.parallel:result is None 2024-06-15T00:53:39.383 INFO:tasks.cephadm.ceph_manager.ceph:waiting for clean 2024-06-15T00:53:39.383 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-15T00:53:39.640 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:40 smithi143 ceph-mon[35860]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:40.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:40.661 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:40 smithi105 ceph-mon[28877]: pgmap v156: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:40.661 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:41.133 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 -- 172.21.15.105:0/278412490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 msgr2=0x7f67d8106350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:41.133 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/278412490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d8106350 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 crypto rx=0x7f67cc004900 tx=0x7f67cc009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 -- 172.21.15.105:0/278412490 shutdown_connections 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/278412490 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d8106350 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/278412490 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 0x7f67d80fd6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 -- 172.21.15.105:0/278412490 >> 172.21.15.105:0/278412490 conn(0x7f67d80f6dc0 msgr2=0x7f67d80f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 -- 172.21.15.105:0/278412490 shutdown_connections 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.132+0000 7f67ddbdc640 1 -- 172.21.15.105:0/278412490 wait complete. 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.133+0000 7f67ddbdc640 1 Processor -- start 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.133+0000 7f67ddbdc640 1 -- start start 2024-06-15T00:53:41.134 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.133+0000 7f67ddbdc640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 0x7f67d8112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:41.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67ddbdc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d81134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:41.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67ddbdc640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8113af0 con 0x7f67d80fdc80 2024-06-15T00:53:41.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67ddbdc640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67d8113c60 con 0x7f67d80fd2e0 2024-06-15T00:53:41.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d81134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:41.136 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d81134f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:43144/0 (socket says 172.21.15.105:43144) 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 -- 172.21.15.105:0/1499718456 learned_addr learned my addr 172.21.15.105:0/1499718456 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d77fe640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 0x7f67d8112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 -- 172.21.15.105:0/1499718456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 msgr2=0x7f67d8112fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 0x7f67d8112fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 -- 172.21.15.105:0/1499718456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67cc005040 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d77fe640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 0x7f67d8112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67d6ffd640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d81134f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f67cc004960 tx=0x7f67cc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67cc038070 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67d810d080 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67cc02ae90 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f67cc009550 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.134+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67d810d5a0 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.135+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67d8066030 con 0x7f67d80fdc80 2024-06-15T00:53:41.137 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.136+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f67cc030020 con 0x7f67d80fdc80 2024-06-15T00:53:41.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.136+0000 7f67bffff640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f67c0065430 0x7f67c00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:41.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.136+0000 7f67d77fe640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f67c0065430 0x7f67c00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:41.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.136+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f67cc0a29c0 con 0x7f67d80fdc80 2024-06-15T00:53:41.138 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.137+0000 7f67d77fe640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f67c0065430 0x7f67c00678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f67c8004d80 tx=0x7f67c800b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:41.141 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.141+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f67cc070e70 con 0x7f67d80fdc80 2024-06-15T00:53:41.301 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.301+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f67d804b480 con 0x7f67c0065430 2024-06-15T00:53:41.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.302+0000 7f67bffff640 1 -- 172.21.15.105:0/1499718456 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27779 (secure 0 0 0) 0x7f67d804b480 con 0x7f67c0065430 2024-06-15T00:53:41.305 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:41.306 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f67c0065430 msgr2=0x7f67c00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f67c0065430 0x7f67c00678e0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 crypto rx=0x7f67c8004d80 tx=0x7f67c800b040 comp rx=0 tx=0).stop 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 msgr2=0x7f67d81134f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d81134f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 crypto rx=0x7f67cc004960 tx=0x7f67cc009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 shutdown_connections 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f67c0065430 0x7f67c00678e0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f67d80fdc80 0x7f67d81134f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 --2- 172.21.15.105:0/1499718456 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f67d80fd2e0 0x7f67d8112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 >> 172.21.15.105:0/1499718456 conn(0x7f67d80f6dc0 msgr2=0x7f67d80f8390 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 shutdown_connections 2024-06-15T00:53:41.307 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:41.304+0000 7f67ddbdc640 1 -- 172.21.15.105:0/1499718456 wait complete. 2024-06-15T00:53:41.911 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":157,"stamp":"2024-06-15T00:53:40.726175+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788084,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682828,"statfs":{"total":575970213888,"available":574139215872,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002111"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-15T00:52:48.405553+0000","last_change":"2024-06-15T00:52:23.426429+0000","last_active":"2024-06-15T00:52:48.405553+0000","last_peered":"2024-06-15T00:52:48.405553+0000","last_clean":"2024-06-15T00:52:48.405553+0000","last_became_active":"2024-06-15T00:52:23.426130+0000","last_became_peered":"2024-06-15T00:52:23.426130+0000","last_unstale":"2024-06-15T00:52:48.405553+0000","last_undegraded":"2024-06-15T00:52:48.405553+0000","last_fullsized":"2024-06-15T00:52:48.405553+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T00:51:37.338898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T00:51:37.338898+0000","last_clean_scrub_stamp":"2024-06-15T00:51:37.338898+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:01:35.024644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.501},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.438}]},{"osd":1,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Jun 15 00:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jun 15 00:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.441}]},{"osd":4,"last update":"Sat Jun 15 00:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378652,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.34799999999999998,"5min":0.34799999999999998,"15min":0.34799999999999998},"min":{"1min":0.24299999999999999,"5min":0.24299999999999999,"15min":0.24299999999999999},"max":{"1min":0.47799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.377},{"interface":"front","average":{"1min":0.34399999999999997,"5min":0.34399999999999997,"15min":0.34399999999999997},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.41699999999999998}]},{"osd":1,"last update":"Sat Jun 15 00:52:36 2024","interfaces":[{"interface":"back","average":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"min":{"1min":0.23899999999999999,"5min":0.23899999999999999,"15min":0.23899999999999999},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.39100000000000001},{"interface":"front","average":{"1min":0.371,"5min":0.371,"15min":0.371},"min":{"1min":0.22600000000000001,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.53800000000000003,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.434}]},{"osd":3,"last update":"Sat Jun 15 00:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.46999999999999997},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.45500000000000002}]},{"osd":4,"last update":"Sat Jun 15 00:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.48399999999999999},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.497}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.51200000000000001}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.29999999999999999,"15min":0.29799999999999999},"min":{"1min":0.20499999999999999,"5min":0.157,"15min":0.157},"max":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.307,"15min":0.30299999999999999},"min":{"1min":0.219,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.436},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Jun 15 00:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Jun 15 00:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":9,"seq":38654705702,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 15 00:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.314,"15min":0.316},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.26500000000000001},{"interface":"front","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.28299999999999997}]},{"osd":2,"last update":"Sat Jun 15 00:52:40 2024","interfaces":[{"interface":"back","average":{"1min":0.379,"5min":0.379,"15min":0.379},"min":{"1min":0.28100000000000003,"5min":0.28100000000000003,"15min":0.28100000000000003},"max":{"1min":0.497,"5min":0.497,"15min":0.497},"last":0.29999999999999999},{"interface":"front","average":{"1min":0.40200000000000002,"5min":0.40200000000000002,"15min":0.40200000000000002},"min":{"1min":0.29499999999999998,"5min":0.29499999999999998,"15min":0.29499999999999998},"max":{"1min":0.58799999999999997,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.32100000000000001}]},{"osd":3,"last update":"Sat Jun 15 00:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.44800000000000001},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.51600000000000001}]},{"osd":4,"last update":"Sat Jun 15 00:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.55300000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.56399999999999995}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.48799999999999999}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T00:53:41.912 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph pg dump --format=json 2024-06-15T00:53:42.167 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:42.613 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:42 smithi105 ceph-mon[28877]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:42.613 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:42 smithi105 ceph-mon[28877]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:53:42.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:42 smithi143 ceph-mon[35860]: pgmap v157: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:42.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:42 smithi143 ceph-mon[35860]: from='client.14420 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:53:43.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 -- 172.21.15.105:0/2370216483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa990102c60 msgr2=0x7fa9900fc2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:43.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 --2- 172.21.15.105:0/2370216483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa990102c60 0x7fa9900fc2f0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 crypto rx=0x7fa9840048d0 tx=0x7fa984009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 -- 172.21.15.105:0/2370216483 shutdown_connections 2024-06-15T00:53:43.820 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 --2- 172.21.15.105:0/2370216483 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa990102c60 0x7fa9900fc2f0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 --2- 172.21.15.105:0/2370216483 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa9901022c0 0x7fa990102690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 -- 172.21.15.105:0/2370216483 >> 172.21.15.105:0/2370216483 conn(0x7fa9900f7e50 msgr2=0x7fa9900fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 -- 172.21.15.105:0/2370216483 shutdown_connections 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.787+0000 7fa9959db640 1 -- 172.21.15.105:0/2370216483 wait complete. 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa9959db640 1 Processor -- start 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa9959db640 1 -- start start 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa9959db640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 0x7fa990113270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa9959db640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa990102c60 0x7fa9901137b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa9959db640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa990113db0 con 0x7fa9901022c0 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa9959db640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa99010d1f0 con 0x7fa990102c60 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 0x7fa990113270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 0x7fa990113270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48620/0 (socket says 172.21.15.105:48620) 2024-06-15T00:53:43.821 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 -- 172.21.15.105:0/2094040648 learned_addr learned my addr 172.21.15.105:0/2094040648 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98e7fc640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa990102c60 0x7fa9901137b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 -- 172.21.15.105:0/2094040648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa990102c60 msgr2=0x7fa9901137b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa990102c60 0x7fa9901137b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 -- 172.21.15.105:0/2094040648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa984005040 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.788+0000 7fa98effd640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 0x7fa990113270 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa980002a50 tx=0x7fa980004600 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.789+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa980007c30 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.789+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa99010d4d0 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.789+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa98000f040 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.789+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa98000daf0 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.789+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa99010da70 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.790+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9900fda30 con 0x7fa9901022c0 2024-06-15T00:53:43.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.790+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa98000dc90 con 0x7fa9901022c0 2024-06-15T00:53:43.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.790+0000 7fa9777fe640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa978065430 0x7fa9780678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:43.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.790+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa980081e70 con 0x7fa9901022c0 2024-06-15T00:53:43.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.790+0000 7fa98e7fc640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa978065430 0x7fa9780678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:43.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.791+0000 7fa98e7fc640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa978065430 0x7fa9780678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa984004540 tx=0x7fa984035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:43.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.794+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa98004cdf0 con 0x7fa9901022c0 2024-06-15T00:53:43.955 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.955+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 --> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fa99004b480 con 0x7fa978065430 2024-06-15T00:53:43.957 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.956+0000 7fa9777fe640 1 -- 172.21.15.105:0/2094040648 <== mgr.14214 v2:172.21.15.105:6800/3392458267 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+27708 (secure 0 0 0) 0x7fa99004b480 con 0x7fa978065430 2024-06-15T00:53:43.958 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:43.958 INFO:teuthology.orchestra.run.smithi105.stderr:dumped all 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa978065430 msgr2=0x7fa9780678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa978065430 0x7fa9780678e0 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 crypto rx=0x7fa984004540 tx=0x7fa984035040 comp rx=0 tx=0).stop 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 msgr2=0x7fa990113270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 0x7fa990113270 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 crypto rx=0x7fa980002a50 tx=0x7fa980004600 comp rx=0 tx=0).stop 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 shutdown_connections 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa978065430 0x7fa9780678e0 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa990102c60 0x7fa9901137b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 --2- 172.21.15.105:0/2094040648 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa9901022c0 0x7fa990113270 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.958+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 >> 172.21.15.105:0/2094040648 conn(0x7fa9900f7e50 msgr2=0x7fa9900f8e50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.959+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 shutdown_connections 2024-06-15T00:53:43.959 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:43.959+0000 7fa9959db640 1 -- 172.21.15.105:0/2094040648 wait complete. 2024-06-15T00:53:44.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:44 smithi143 ceph-mon[35860]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:44.666 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:44 smithi105 ceph-mon[28877]: pgmap v158: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:44.690 INFO:teuthology.orchestra.run.smithi105.stdout:{"pg_ready":true,"pg_map":{"version":158,"stamp":"2024-06-15T00:53:42.726515+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":3,"num_osds":6,"num_per_pool_osds":6,"num_per_pool_omap_osds":3,"kb":562470912,"kb_used":1788084,"kb_used_data":3924,"kb_used_omap":9,"kb_used_meta":1784054,"kb_avail":560682828,"statfs":{"total":575970213888,"available":574139215872,"internally_reserved":0,"allocated":4018176,"data_stored":2624466,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":10224,"internal_metadata":1826871312},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"12.002045"},"pg_stats":[{"pgid":"1.0","version":"24'117","reported_seq":189,"reported_epoch":40,"state":"active+clean","last_fresh":"2024-06-15T00:52:48.405553+0000","last_change":"2024-06-15T00:52:23.426429+0000","last_active":"2024-06-15T00:52:48.405553+0000","last_peered":"2024-06-15T00:52:48.405553+0000","last_clean":"2024-06-15T00:52:48.405553+0000","last_became_active":"2024-06-15T00:52:23.426130+0000","last_became_peered":"2024-06-15T00:52:23.426130+0000","last_unstale":"2024-06-15T00:52:48.405553+0000","last_undegraded":"2024-06-15T00:52:48.405553+0000","last_fullsized":"2024-06-15T00:52:48.405553+0000","mapping_epoch":34,"log_start":"0'0","ondisk_log_start":"0'0","created":21,"last_epoch_clean":35,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2024-06-15T00:51:37.338898+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2024-06-15T00:51:37.338898+0000","last_clean_scrub_stamp":"2024-06-15T00:51:37.338898+0000","objects_scrubbed":0,"log_size":117,"log_dups_size":0,"ondisk_log_size":117,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"last_scrub_duration":0,"scrub_schedule":"periodic scrub scheduled @ 2024-06-16T12:01:35.024644+0000","scrub_duration":0,"objects_trimmed":0,"snaptrim_duration":0,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,0,1],"acting":[3,0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]}],"pool_stats":[{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":590368,"num_objects":2,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":116,"num_read_kb":100,"num_write":159,"num_write_kb":2416,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":2375680,"data_stored":2361472,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":117,"ondisk_log_size":117,"up":3,"acting":3,"num_store_stats":4}],"osd_stats":[{"osd":5,"up_from":40,"seq":171798691854,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297720,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447432,"statfs":{"total":95995035648,"available":95690170368,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,4],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39500000000000002}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.36699999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.33400000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.34999999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.375}]}]},{"osd":4,"up_from":35,"seq":150323855378,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,3,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51600000000000001,"5min":0.51600000000000001,"15min":0.51600000000000001},"min":{"1min":0.26600000000000001,"5min":0.26600000000000001,"15min":0.26600000000000001},"max":{"1min":0.88300000000000001,"5min":0.88300000000000001,"15min":0.88300000000000001},"last":0.501},{"interface":"front","average":{"1min":0.51100000000000001,"5min":0.51100000000000001,"15min":0.51100000000000001},"min":{"1min":0.30499999999999999,"5min":0.30499999999999999,"15min":0.30499999999999999},"max":{"1min":0.78500000000000003,"5min":0.78500000000000003,"15min":0.78500000000000003},"last":0.438}]},{"osd":1,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.51900000000000002,"5min":0.51900000000000002,"15min":0.51900000000000002},"min":{"1min":0.33200000000000002,"5min":0.33200000000000002,"15min":0.33200000000000002},"max":{"1min":0.85799999999999998,"5min":0.85799999999999998,"15min":0.85799999999999998},"last":0.42199999999999999},{"interface":"front","average":{"1min":0.51300000000000001,"5min":0.51300000000000001,"15min":0.51300000000000001},"min":{"1min":0.35399999999999998,"5min":0.35399999999999998,"15min":0.35399999999999998},"max":{"1min":0.72499999999999998,"5min":0.72499999999999998,"15min":0.72499999999999998},"last":0.46500000000000002}]},{"osd":2,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.54500000000000004,"5min":0.54500000000000004,"15min":0.54500000000000004},"min":{"1min":0.40699999999999997,"5min":0.40699999999999997,"15min":0.40699999999999997},"max":{"1min":0.94399999999999995,"5min":0.94399999999999995,"15min":0.94399999999999995},"last":0.40799999999999997},{"interface":"front","average":{"1min":0.54300000000000004,"5min":0.54300000000000004,"15min":0.54300000000000004},"min":{"1min":0.40600000000000003,"5min":0.40600000000000003,"15min":0.40600000000000003},"max":{"1min":0.82199999999999995,"5min":0.82199999999999995,"15min":0.82199999999999995},"last":0.48499999999999999}]},{"osd":3,"last update":"Sat Jun 15 00:53:23 2024","interfaces":[{"interface":"back","average":{"1min":0.42199999999999999,"5min":0.42199999999999999,"15min":0.42199999999999999},"min":{"1min":0.27600000000000002,"5min":0.27600000000000002,"15min":0.27600000000000002},"max":{"1min":0.624,"5min":0.624,"15min":0.624},"last":0.39400000000000002},{"interface":"front","average":{"1min":0.42799999999999999,"5min":0.42799999999999999,"15min":0.42799999999999999},"min":{"1min":0.29299999999999998,"5min":0.29299999999999998,"15min":0.29299999999999998},"max":{"1min":0.66500000000000004,"5min":0.66500000000000004,"15min":0.66500000000000004},"last":0.45200000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.374}]}]},{"osd":3,"up_from":27,"seq":115964117015,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298308,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446844,"statfs":{"total":95995035648,"available":95689568256,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,2,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.40100000000000002,"5min":0.40100000000000002,"15min":0.40100000000000002},"max":{"1min":0.70699999999999996,"5min":0.70699999999999996,"15min":0.70699999999999996},"last":0.48199999999999998},{"interface":"front","average":{"1min":0.51400000000000001,"5min":0.51400000000000001,"15min":0.51400000000000001},"min":{"1min":0.35599999999999998,"5min":0.35599999999999998,"15min":0.35599999999999998},"max":{"1min":0.71699999999999997,"5min":0.71699999999999997,"15min":0.71699999999999997},"last":0.40300000000000002}]},{"osd":1,"last update":"Sat Jun 15 00:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.53200000000000003},"min":{"1min":0.36399999999999999,"5min":0.36399999999999999,"15min":0.36399999999999999},"max":{"1min":1.3089999999999999,"5min":1.3089999999999999,"15min":1.3089999999999999},"last":0.41399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.373,"5min":0.373,"15min":0.373},"max":{"1min":0.67500000000000004,"5min":0.67500000000000004,"15min":0.67500000000000004},"last":0.45600000000000002}]},{"osd":2,"last update":"Sat Jun 15 00:53:00 2024","interfaces":[{"interface":"back","average":{"1min":0.47399999999999998,"5min":0.47399999999999998,"15min":0.47399999999999998},"min":{"1min":0.38200000000000001,"5min":0.38200000000000001,"15min":0.38200000000000001},"max":{"1min":0.69299999999999995,"5min":0.69299999999999995,"15min":0.69299999999999995},"last":0.35499999999999998},{"interface":"front","average":{"1min":0.497,"5min":0.497,"15min":0.497},"min":{"1min":0.35999999999999999,"5min":0.35999999999999999,"15min":0.35999999999999999},"max":{"1min":0.72699999999999998,"5min":0.72699999999999998,"15min":0.72699999999999998},"last":0.441}]},{"osd":4,"last update":"Sat Jun 15 00:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.35299999999999998,"5min":0.35299999999999998,"15min":0.35299999999999998},"min":{"1min":0.224,"5min":0.224,"15min":0.224},"max":{"1min":0.503,"5min":0.503,"15min":0.503},"last":0.32400000000000001},{"interface":"front","average":{"1min":0.36499999999999999,"5min":0.36499999999999999,"15min":0.36499999999999999},"min":{"1min":0.24099999999999999,"5min":0.24099999999999999,"15min":0.24099999999999999},"max":{"1min":0.52100000000000002,"5min":0.52100000000000002,"15min":0.52100000000000002},"last":0.42299999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.38600000000000001}]}]},{"osd":2,"up_from":19,"seq":81604378653,"num_pgs":0,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":0,"kb":93745152,"kb_used":297724,"kb_used_data":364,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93447428,"statfs":{"total":95995035648,"available":95690166272,"internally_reserved":0,"allocated":372736,"data_stored":142227,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,1,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.32300000000000001,"5min":0.34300000000000003,"15min":0.34599999999999997},"min":{"1min":0.222,"5min":0.222,"15min":0.222},"max":{"1min":0.41799999999999998,"5min":0.47799999999999998,"15min":0.47799999999999998},"last":0.22500000000000001},{"interface":"front","average":{"1min":0.32900000000000001,"5min":0.34100000000000003,"15min":0.34300000000000003},"min":{"1min":0.247,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.46300000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"last":0.26700000000000002}]},{"osd":1,"last update":"Sat Jun 15 00:53:37 2024","interfaces":[{"interface":"back","average":{"1min":0.33100000000000002,"5min":0.35399999999999998,"15min":0.35799999999999998},"min":{"1min":0.23400000000000001,"5min":0.23400000000000001,"15min":0.23400000000000001},"max":{"1min":0.441,"5min":0.497,"15min":0.497},"last":0.23699999999999999},{"interface":"front","average":{"1min":0.35199999999999998,"5min":0.36699999999999999,"15min":0.36899999999999999},"min":{"1min":0.245,"5min":0.22600000000000001,"15min":0.22600000000000001},"max":{"1min":0.48299999999999998,"5min":0.53800000000000003,"15min":0.53800000000000003},"last":0.254}]},{"osd":3,"last update":"Sat Jun 15 00:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.46600000000000003,"5min":0.46600000000000003,"15min":0.46600000000000003},"min":{"1min":0.309,"5min":0.309,"15min":0.309},"max":{"1min":0.60299999999999998,"5min":0.60299999999999998,"15min":0.60299999999999998},"last":0.35399999999999998},{"interface":"front","average":{"1min":0.47699999999999998,"5min":0.47699999999999998,"15min":0.47699999999999998},"min":{"1min":0.29599999999999999,"5min":0.29599999999999999,"15min":0.29599999999999999},"max":{"1min":0.70599999999999996,"5min":0.70599999999999996,"15min":0.70599999999999996},"last":0.433}]},{"osd":4,"last update":"Sat Jun 15 00:53:24 2024","interfaces":[{"interface":"back","average":{"1min":0.53500000000000003,"5min":0.53500000000000003,"15min":0.53500000000000003},"min":{"1min":0.40400000000000003,"5min":0.40400000000000003,"15min":0.40400000000000003},"max":{"1min":0.76300000000000001,"5min":0.76300000000000001,"15min":0.76300000000000001},"last":0.34300000000000003},{"interface":"front","average":{"1min":0.52200000000000002,"5min":0.52200000000000002,"15min":0.52200000000000002},"min":{"1min":0.39200000000000002,"5min":0.39200000000000002,"15min":0.39200000000000002},"max":{"1min":0.66600000000000004,"5min":0.66600000000000004,"15min":0.66600000000000004},"last":0.42099999999999999}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.39800000000000002}]}]},{"osd":1,"up_from":14,"seq":60129542177,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[0,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Sat Jun 15 00:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.316,"5min":0.29999999999999999,"15min":0.29799999999999999},"min":{"1min":0.20499999999999999,"5min":0.157,"15min":0.157},"max":{"1min":0.45000000000000001,"5min":0.45000000000000001,"15min":0.45000000000000001},"last":0.40200000000000002},{"interface":"front","average":{"1min":0.33000000000000002,"5min":0.307,"15min":0.30299999999999999},"min":{"1min":0.219,"5min":0.19700000000000001,"15min":0.19700000000000001},"max":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"last":0.44900000000000001}]},{"osd":2,"last update":"Sat Jun 15 00:52:38 2024","interfaces":[{"interface":"back","average":{"1min":0.372,"5min":0.372,"15min":0.372},"min":{"1min":0.186,"5min":0.186,"15min":0.186},"max":{"1min":0.501,"5min":0.501,"15min":0.501},"last":0.436},{"interface":"front","average":{"1min":0.373,"5min":0.373,"15min":0.373},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"last":0.48899999999999999}]},{"osd":3,"last update":"Sat Jun 15 00:52:59 2024","interfaces":[{"interface":"back","average":{"1min":0.501,"5min":0.501,"15min":0.501},"min":{"1min":0.374,"5min":0.374,"15min":0.374},"max":{"1min":0.60799999999999998,"5min":0.60799999999999998,"15min":0.60799999999999998},"last":0.57599999999999996},{"interface":"front","average":{"1min":0.48699999999999999,"5min":0.48699999999999999,"15min":0.48699999999999999},"min":{"1min":0.35099999999999998,"5min":0.35099999999999998,"15min":0.35099999999999998},"max":{"1min":0.68700000000000006,"5min":0.68700000000000006,"15min":0.68700000000000006},"last":0.64400000000000002}]},{"osd":4,"last update":"Sat Jun 15 00:53:25 2024","interfaces":[{"interface":"back","average":{"1min":0.52700000000000002,"5min":0.52700000000000002,"15min":0.52700000000000002},"min":{"1min":0.38900000000000001,"5min":0.38900000000000001,"15min":0.38900000000000001},"max":{"1min":0.85599999999999998,"5min":0.85599999999999998,"15min":0.85599999999999998},"last":0.63300000000000001},{"interface":"front","average":{"1min":0.54700000000000004,"5min":0.54700000000000004,"15min":0.54700000000000004},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.81399999999999995,"5min":0.81399999999999995,"15min":0.81399999999999995},"last":0.65800000000000003}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.622}]}]},{"osd":0,"up_from":9,"seq":38654705703,"num_pgs":1,"num_osds":1,"num_per_pool_osds":1,"num_per_pool_omap_osds":1,"kb":93745152,"kb_used":298304,"kb_used_data":944,"kb_used_omap":1,"kb_used_meta":297342,"kb_avail":93446848,"statfs":{"total":95995035648,"available":95689572352,"internally_reserved":0,"allocated":966656,"data_stored":732595,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":1704,"internal_metadata":304478552},"hb_peers":[1,2,3,4,5],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Sat Jun 15 00:53:13 2024","interfaces":[{"interface":"back","average":{"1min":0.30499999999999999,"5min":0.314,"15min":0.316},"min":{"1min":0.185,"5min":0.185,"15min":0.185},"max":{"1min":0.61299999999999999,"5min":0.61299999999999999,"15min":0.61299999999999999},"last":0.23599999999999999},{"interface":"front","average":{"1min":0.315,"5min":0.32300000000000001,"15min":0.32500000000000001},"min":{"1min":0.214,"5min":0.214,"15min":0.214},"max":{"1min":0.65000000000000002,"5min":0.65000000000000002,"15min":0.65000000000000002},"last":0.28399999999999997}]},{"osd":2,"last update":"Sat Jun 15 00:53:40 2024","interfaces":[{"interface":"back","average":{"1min":0.308,"5min":0.36399999999999999,"15min":0.374},"min":{"1min":0.245,"5min":0.245,"15min":0.245},"max":{"1min":0.44,"5min":0.497,"15min":0.497},"last":0.30599999999999999},{"interface":"front","average":{"1min":0.32800000000000001,"5min":0.38700000000000001,"15min":0.39700000000000002},"min":{"1min":0.26700000000000002,"5min":0.26700000000000002,"15min":0.26700000000000002},"max":{"1min":0.45400000000000001,"5min":0.58799999999999997,"15min":0.58799999999999997},"last":0.313}]},{"osd":3,"last update":"Sat Jun 15 00:53:01 2024","interfaces":[{"interface":"back","average":{"1min":0.48299999999999998,"5min":0.48299999999999998,"15min":0.48299999999999998},"min":{"1min":0.37,"5min":0.37,"15min":0.37},"max":{"1min":0.69499999999999995,"5min":0.69499999999999995,"15min":0.69499999999999995},"last":0.78800000000000003},{"interface":"front","average":{"1min":0.48899999999999999,"5min":0.48899999999999999,"15min":0.48899999999999999},"min":{"1min":0.38300000000000001,"5min":0.38300000000000001,"15min":0.38300000000000001},"max":{"1min":0.629,"5min":0.629,"15min":0.629},"last":0.66300000000000003}]},{"osd":4,"last update":"Sat Jun 15 00:53:26 2024","interfaces":[{"interface":"back","average":{"1min":0.498,"5min":0.498,"15min":0.498},"min":{"1min":0.40000000000000002,"5min":0.40000000000000002,"15min":0.40000000000000002},"max":{"1min":0.72199999999999998,"5min":0.72199999999999998,"15min":0.72199999999999998},"last":0.68400000000000005},{"interface":"front","average":{"1min":0.51200000000000001,"5min":0.51200000000000001,"15min":0.51200000000000001},"min":{"1min":0.39500000000000002,"5min":0.39500000000000002,"15min":0.39500000000000002},"max":{"1min":0.67800000000000005,"5min":0.67800000000000005,"15min":0.67800000000000005},"last":0.70399999999999996}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.745}]}]}],"pool_statfs":[{"poolid":1,"osd":0,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":1,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":2,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},{"poolid":1,"osd":3,"total":0,"available":0,"internally_reserved":0,"allocated":593920,"data_stored":590368,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0}]}} 2024-06-15T00:53:44.691 INFO:tasks.cephadm.ceph_manager.ceph:clean! 2024-06-15T00:53:44.691 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2024-06-15T00:53:44.691 INFO:tasks.cephadm.ceph_manager.ceph:wait_until_healthy 2024-06-15T00:53:44.691 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:53:44.942 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:45.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:45 smithi143 ceph-mon[35860]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:53:45.776 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:45 smithi105 ceph-mon[28877]: from='client.14424 -' entity='client.admin' cmd=[{"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-06-15T00:53:46.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:46 smithi143 ceph-mon[35860]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:46.659 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:46 smithi105 ceph-mon[28877]: pgmap v159: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:46.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 -- 172.21.15.105:0/929001492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 msgr2=0x7fdbdc0fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:46.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/929001492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc0fb1f0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 crypto rx=0x7fdbd00048d0 tx=0x7fdbd0009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 -- 172.21.15.105:0/929001492 shutdown_connections 2024-06-15T00:53:46.660 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/929001492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc0fb1f0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/929001492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 0x7fdbdc101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 -- 172.21.15.105:0/929001492 >> 172.21.15.105:0/929001492 conn(0x7fdbdc0f6d70 msgr2=0x7fdbdc0f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:46.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 -- 172.21.15.105:0/929001492 shutdown_connections 2024-06-15T00:53:46.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.660+0000 7fdbe14ff640 1 -- 172.21.15.105:0/929001492 wait complete. 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbe14ff640 1 Processor -- start 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbe14ff640 1 -- start start 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbe14ff640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 0x7fdbdc10df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbe14ff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc10e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbe14ff640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc10eb80 con 0x7fdbdc101b60 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbe14ff640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbdc10ecf0 con 0x7fdbdc1011c0 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbda7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc10e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbda7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc10e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48628/0 (socket says 172.21.15.105:48628) 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbda7fc640 1 -- 172.21.15.105:0/2466802987 learned_addr learned my addr 172.21.15.105:0/2466802987 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbdaffd640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 0x7fdbdc10df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbda7fc640 1 -- 172.21.15.105:0/2466802987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 msgr2=0x7fdbdc10df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbda7fc640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 0x7fdbdc10df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.663 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbda7fc640 1 -- 172.21.15.105:0/2466802987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbd0005040 con 0x7fdbdc101b60 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.661+0000 7fdbdaffd640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 0x7fdbdc10df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.662+0000 7fdbda7fc640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc10e4a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0000c00 tx=0x7fdbd0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.662+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbd0038070 con 0x7fdbdc101b60 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.662+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbd00096f0 con 0x7fdbdc101b60 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.662+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbdc116fd0 con 0x7fdbdc101b60 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.662+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdbd0032da0 con 0x7fdbdc101b60 2024-06-15T00:53:46.664 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.662+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbdc117410 con 0x7fdbdc101b60 2024-06-15T00:53:46.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.663+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbdc0fc930 con 0x7fdbdc101b60 2024-06-15T00:53:46.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.663+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdbd0030020 con 0x7fdbdc101b60 2024-06-15T00:53:46.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.664+0000 7fdbc37fe640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdbc4065430 0x7fdbc40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:46.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.664+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdbd00a2b30 con 0x7fdbdc101b60 2024-06-15T00:53:46.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.664+0000 7fdbdaffd640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdbc4065430 0x7fdbc40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:46.665 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.664+0000 7fdbdaffd640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdbc4065430 0x7fdbc40678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc006fd0 tx=0x7fdbcc008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:46.668 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.668+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdbd0070f80 con 0x7fdbdc101b60 2024-06-15T00:53:46.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.865+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdbdc0faa20 con 0x7fdbdc101b60 2024-06-15T00:53:46.866 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.866+0000 7fdbc37fe640 1 -- 172.21.15.105:0/2466802987 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdbdc0faa20 con 0x7fdbdc101b60 2024-06-15T00:53:46.866 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:46.866 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdbc4065430 msgr2=0x7fdbc40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdbc4065430 0x7fdbc40678e0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 crypto rx=0x7fdbcc006fd0 tx=0x7fdbcc008040 comp rx=0 tx=0).stop 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 msgr2=0x7fdbdc10e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc10e4a0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 crypto rx=0x7fdbd0000c00 tx=0x7fdbd0009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 shutdown_connections 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdbc4065430 0x7fdbc40678e0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdbdc101b60 0x7fdbdc10e4a0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.867+0000 7fdbe14ff640 1 --2- 172.21.15.105:0/2466802987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdbdc1011c0 0x7fdbdc10df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.868+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 >> 172.21.15.105:0/2466802987 conn(0x7fdbdc0f6d70 msgr2=0x7fdbdc0f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.868+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 shutdown_connections 2024-06-15T00:53:46.868 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:46.868+0000 7fdbe14ff640 1 -- 172.21.15.105:0/2466802987 wait complete. 2024-06-15T00:53:47.571 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:47 smithi105 ceph-mon[28877]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:47.571 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:47 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2466802987' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:53:47.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:47 smithi143 ceph-mon[35860]: pgmap v160: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:47.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2466802987' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:53:50.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:49 smithi143 ceph-mon[35860]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:50.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:49 smithi105 ceph-mon[28877]: pgmap v161: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:50.572 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:53:50.827 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:50.949 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:53:50.949 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:53:50.949 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:53:50.949 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:53:51.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:53:51.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:53:51.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:53:51.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:53:52.046 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:51 smithi105 ceph-mon[28877]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:52.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:51 smithi143 ceph-mon[35860]: pgmap v162: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:52.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 -- 172.21.15.105:0/3371392944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 msgr2=0x7f3ac40fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:52.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 --2- 172.21.15.105:0/3371392944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac40fb940 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40048d0 tx=0x7f3ab4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 -- 172.21.15.105:0/3371392944 shutdown_connections 2024-06-15T00:53:52.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 --2- 172.21.15.105:0/3371392944 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac40fbe80 0x7f3ac40fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 --2- 172.21.15.105:0/3371392944 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac40fb940 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 -- 172.21.15.105:0/3371392944 >> 172.21.15.105:0/3371392944 conn(0x7f3ac4072110 msgr2=0x7f3ac4072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 -- 172.21.15.105:0/3371392944 shutdown_connections 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.517+0000 7f3ac988d640 1 -- 172.21.15.105:0/3371392944 wait complete. 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac988d640 1 Processor -- start 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac988d640 1 -- start start 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac988d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac410e0a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac988d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac40fbe80 0x7f3ac410e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac988d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac410eb20 con 0x7f3ac40fb570 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac988d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ac410ec90 con 0x7f3ac40fbe80 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac410e0a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac410e0a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48644/0 (socket says 172.21.15.105:48644) 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 -- 172.21.15.105:0/1377000320 learned_addr learned my addr 172.21.15.105:0/1377000320 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:52.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 -- 172.21.15.105:0/1377000320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac40fbe80 msgr2=0x7f3ac410e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac27fc640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac40fbe80 0x7f3ac410e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac40fbe80 0x7f3ac410e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 -- 172.21.15.105:0/1377000320 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab4005040 con 0x7f3ac40fb570 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.518+0000 7f3ac2ffd640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac410e0a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40048a0 tx=0x7f3ab4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.519+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab4038070 con 0x7f3ac40fb570 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.519+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ab4032d30 con 0x7f3ac40fb570 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.519+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ac4116f70 con 0x7f3ac40fb570 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.519+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f3ab4009770 con 0x7f3ac40fb570 2024-06-15T00:53:52.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.519+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ac4117400 con 0x7f3ac40fb570 2024-06-15T00:53:52.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.520+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ac40fc920 con 0x7f3ac40fb570 2024-06-15T00:53:52.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.520+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f3ab4030020 con 0x7f3ac40fb570 2024-06-15T00:53:52.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.521+0000 7f3aab7fe640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f3aac065430 0x7f3aac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:52.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.521+0000 7f3ac27fc640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f3aac065430 0x7f3aac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:52.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.521+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f3ab40a2310 con 0x7f3ac40fb570 2024-06-15T00:53:52.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.521+0000 7f3ac27fc640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f3aac065430 0x7f3aac0678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4065740 tx=0x7f3ab8008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:52.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.525+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f3ab40707c0 con 0x7f3ac40fb570 2024-06-15T00:53:52.724 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.724+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f3ac4072c20 con 0x7f3ac40fb570 2024-06-15T00:53:52.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.724+0000 7f3aab7fe640 1 -- 172.21.15.105:0/1377000320 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f3ab4073de0 con 0x7f3ac40fb570 2024-06-15T00:53:52.725 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:52.725 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:53:52.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f3aac065430 msgr2=0x7f3aac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:52.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f3aac065430 0x7f3aac0678e0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 crypto rx=0x7f3ac4065740 tx=0x7f3ab8008040 comp rx=0 tx=0).stop 2024-06-15T00:53:52.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 msgr2=0x7f3ac410e0a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:52.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac410e0a0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 crypto rx=0x7f3ab40048a0 tx=0x7f3ab4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 shutdown_connections 2024-06-15T00:53:52.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f3aac065430 0x7f3aac0678e0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f3ac40fbe80 0x7f3ac410e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 --2- 172.21.15.105:0/1377000320 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f3ac40fb570 0x7f3ac410e0a0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:52.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 >> 172.21.15.105:0/1377000320 conn(0x7f3ac4072110 msgr2=0x7f3ac40f8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:52.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 shutdown_connections 2024-06-15T00:53:52.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:52.727+0000 7f3ac988d640 1 -- 172.21.15.105:0/1377000320 wait complete. 2024-06-15T00:53:53.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:52 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1377000320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:53:53.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:52 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1377000320' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:53:54.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:53 smithi143 ceph-mon[35860]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:54.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:53 smithi105 ceph-mon[28877]: pgmap v163: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:55.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:55.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:53:56.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:55 smithi143 ceph-mon[35860]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:55 smithi105 ceph-mon[28877]: pgmap v164: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:56.489 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:53:56.751 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:53:58.078 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:57 smithi105 ceph-mon[28877]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:58.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:57 smithi143 ceph-mon[35860]: pgmap v165: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 -- 172.21.15.105:0/3720082444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 msgr2=0x7f0cbc102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/3720082444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc102670 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 crypto rx=0x7f0cb40048d0 tx=0x7f0cb4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 -- 172.21.15.105:0/3720082444 shutdown_connections 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/3720082444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cbc102c40 0x7f0cbc0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/3720082444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc102670 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 -- 172.21.15.105:0/3720082444 >> 172.21.15.105:0/3720082444 conn(0x7f0cbc0f7e50 msgr2=0x7f0cbc0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:58.442 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 -- 172.21.15.105:0/3720082444 shutdown_connections 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 -- 172.21.15.105:0/3720082444 wait complete. 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.440+0000 7f0cc0d32640 1 Processor -- start 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cc0d32640 1 -- start start 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cc0d32640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc112ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cc0d32640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cbc102c40 0x7f0cbc113420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cc0d32640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc113a20 con 0x7f0cbc1022a0 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cc0d32640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cbc10ce60 con 0x7f0cbc102c40 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc112ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc112ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52880/0 (socket says 172.21.15.105:52880) 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 -- 172.21.15.105:0/2436649697 learned_addr learned my addr 172.21.15.105:0/2436649697 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cb9d9b640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cbc102c40 0x7f0cbc113420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:58.443 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 -- 172.21.15.105:0/2436649697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cbc102c40 msgr2=0x7f0cbc113420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cbc102c40 0x7f0cbc113420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 -- 172.21.15.105:0/2436649697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb4005040 con 0x7f0cbc1022a0 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cba59c640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc112ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4000c00 tx=0x7f0cb4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cb4038070 con 0x7f0cbc1022a0 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cb40096f0 con 0x7f0cbc1022a0 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cbc10d0e0 con 0x7f0cbc1022a0 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.441+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f0cb4032da0 con 0x7f0cbc1022a0 2024-06-15T00:53:58.444 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.442+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cbc10d620 con 0x7f0cbc1022a0 2024-06-15T00:53:58.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.443+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cbc0fda10 con 0x7f0cbc1022a0 2024-06-15T00:53:58.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.443+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f0cb4030020 con 0x7f0cbc1022a0 2024-06-15T00:53:58.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.443+0000 7f0caaffd640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0ca4065430 0x7f0ca40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:53:58.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.444+0000 7f0cb9d9b640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0ca4065430 0x7f0ca40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:53:58.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.444+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f0cb40a2ab0 con 0x7f0cbc1022a0 2024-06-15T00:53:58.445 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.444+0000 7f0cb9d9b640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0ca4065430 0x7f0ca40678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc10e7a0 tx=0x7f0cb0009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:53:58.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.448+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f0cb4070f60 con 0x7f0cbc1022a0 2024-06-15T00:53:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.645+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f0cbc1092d0 con 0x7f0cbc1022a0 2024-06-15T00:53:58.646 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.646+0000 7f0caaffd640 1 -- 172.21.15.105:0/2436649697 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f0cb4074580 con 0x7f0cbc1022a0 2024-06-15T00:53:58.646 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:53:58.646 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0ca4065430 msgr2=0x7f0ca40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0ca4065430 0x7f0ca40678e0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 crypto rx=0x7f0cbc10e7a0 tx=0x7f0cb0009040 comp rx=0 tx=0).stop 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 msgr2=0x7f0cbc112ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc112ee0 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 crypto rx=0x7f0cb4000c00 tx=0x7f0cb4009da0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 shutdown_connections 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f0ca4065430 0x7f0ca40678e0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.648 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f0cbc102c40 0x7f0cbc113420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 --2- 172.21.15.105:0/2436649697 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f0cbc1022a0 0x7f0cbc112ee0 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:53:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 >> 172.21.15.105:0/2436649697 conn(0x7f0cbc0f7e50 msgr2=0x7f0cbc0f8e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:53:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 shutdown_connections 2024-06-15T00:53:58.649 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:53:58.648+0000 7f0cc0d32640 1 -- 172.21.15.105:0/2436649697 wait complete. 2024-06-15T00:53:58.992 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:58 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2436649697' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:53:59.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:58 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2436649697' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:00.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:53:59 smithi143 ceph-mon[35860]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:00.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:53:59 smithi105 ceph-mon[28877]: pgmap v166: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:02.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:01 smithi143 ceph-mon[35860]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:02.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:01 smithi105 ceph-mon[28877]: pgmap v167: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:02.300 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:02.566 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:03.936 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:03 smithi105 ceph-mon[28877]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:04.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:03 smithi143 ceph-mon[35860]: pgmap v168: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:04.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 -- 172.21.15.105:0/355380788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24041022a0 msgr2=0x7f2404102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:04.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 --2- 172.21.15.105:0/355380788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24041022a0 0x7f2404102670 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 crypto rx=0x7f24000048d0 tx=0x7f2400009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 -- 172.21.15.105:0/355380788 shutdown_connections 2024-06-15T00:54:04.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 --2- 172.21.15.105:0/355380788 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2404102c40 0x7f24040fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 --2- 172.21.15.105:0/355380788 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f24041022a0 0x7f2404102670 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 -- 172.21.15.105:0/355380788 >> 172.21.15.105:0/355380788 conn(0x7f24040f7e70 msgr2=0x7f24040fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 -- 172.21.15.105:0/355380788 shutdown_connections 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 -- 172.21.15.105:0/355380788 wait complete. 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.153+0000 7f240b73c640 1 Processor -- start 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f240b73c640 1 -- start start 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f240b73c640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24041022a0 0x7f2404112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f240b73c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 0x7f2404113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f240b73c640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2404113b90 con 0x7f2404102c40 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f240b73c640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f240410cf80 con 0x7f24041022a0 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 0x7f2404113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:04.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 0x7f2404113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35318/0 (socket says 172.21.15.105:35318) 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 -- 172.21.15.105:0/2282902842 learned_addr learned my addr 172.21.15.105:0/2282902842 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f24094d8640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24041022a0 0x7f2404112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 -- 172.21.15.105:0/2282902842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24041022a0 msgr2=0x7f2404112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24041022a0 0x7f2404112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 -- 172.21.15.105:0/2282902842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2400005040 con 0x7f2404102c40 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f24094d8640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24041022a0 0x7f2404112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f2408cd7640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 0x7f2404113500 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f23f400b790 tx=0x7f23f40091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23f400c700 con 0x7f2404102c40 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f240410d260 con 0x7f2404102c40 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f23f400cd10 con 0x7f2404102c40 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.154+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f23f4013370 con 0x7f2404102c40 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.155+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f240410d780 con 0x7f2404102c40 2024-06-15T00:54:04.171 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.156+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24040fda10 con 0x7f2404102c40 2024-06-15T00:54:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.156+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f23f4012070 con 0x7f2404102c40 2024-06-15T00:54:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.156+0000 7f23f9ffb640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f23f0065430 0x7f23f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.156+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f23f4082140 con 0x7f2404102c40 2024-06-15T00:54:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.156+0000 7f24094d8640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f23f0065430 0x7f23f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.157+0000 7f24094d8640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f23f0065430 0x7f23f00678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2400004540 tx=0x7f2400035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:04.172 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.160+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f23f40505f0 con 0x7f2404102c40 2024-06-15T00:54:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.360+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f24040fba90 con 0x7f2404102c40 2024-06-15T00:54:04.360 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.360+0000 7f23f9ffb640 1 -- 172.21.15.105:0/2282902842 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f23f4053c10 con 0x7f2404102c40 2024-06-15T00:54:04.360 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:04.361 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f23f0065430 msgr2=0x7f23f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f23f0065430 0x7f23f00678e0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 crypto rx=0x7f2400004540 tx=0x7f2400035040 comp rx=0 tx=0).stop 2024-06-15T00:54:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 msgr2=0x7f2404113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 0x7f2404113500 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 crypto rx=0x7f23f400b790 tx=0x7f23f40091d0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 shutdown_connections 2024-06-15T00:54:04.362 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f23f0065430 0x7f23f00678e0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2404102c40 0x7f2404113500 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 --2- 172.21.15.105:0/2282902842 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f24041022a0 0x7f2404112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:04.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 >> 172.21.15.105:0/2282902842 conn(0x7f24040f7e70 msgr2=0x7f24040f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:04.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 shutdown_connections 2024-06-15T00:54:04.363 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:04.362+0000 7f240b73c640 1 -- 172.21.15.105:0/2282902842 wait complete. 2024-06-15T00:54:05.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:04 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2282902842' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:05.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:04 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2282902842' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:06.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:05 smithi143 ceph-mon[35860]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:06.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:05 smithi105 ceph-mon[28877]: pgmap v169: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:08.083 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:08.110 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:07 smithi105 ceph-mon[28877]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:08.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:07 smithi143 ceph-mon[35860]: pgmap v170: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:08.349 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:09.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 -- 172.21.15.105:0/2488404358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 msgr2=0x7fd7fc065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:09.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 --2- 172.21.15.105:0/2488404358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc065470 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec009c50 tx=0x7fd7ec004d10 comp rx=0 tx=0).stop 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 -- 172.21.15.105:0/2488404358 shutdown_connections 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 --2- 172.21.15.105:0/2488404358 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0fdeb0 0x7fd7fc0fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 --2- 172.21.15.105:0/2488404358 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc065470 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 -- 172.21.15.105:0/2488404358 >> 172.21.15.105:0/2488404358 conn(0x7fd7fc072110 msgr2=0x7fd7fc072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 -- 172.21.15.105:0/2488404358 shutdown_connections 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.983+0000 7fd803f35640 1 -- 172.21.15.105:0/2488404358 wait complete. 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd803f35640 1 Processor -- start 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd803f35640 1 -- start start 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd803f35640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd803f35640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0fdeb0 0x7fd7fc113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd803f35640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc113b90 con 0x7fd7fc0650a0 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd803f35640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7fc10cf40 con 0x7fd7fc0fdeb0 2024-06-15T00:54:09.986 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd801cd1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd801cd1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35330/0 (socket says 172.21.15.105:35330) 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd801cd1640 1 -- 172.21.15.105:0/2505907617 learned_addr learned my addr 172.21.15.105:0/2505907617 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd8014d0640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0fdeb0 0x7fd7fc113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd801cd1640 1 -- 172.21.15.105:0/2505907617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0fdeb0 msgr2=0x7fd7fc113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd801cd1640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0fdeb0 0x7fd7fc113500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.984+0000 7fd801cd1640 1 -- 172.21.15.105:0/2505907617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7ec0098b0 con 0x7fd7fc0650a0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.985+0000 7fd801cd1640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc112fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec003400 tx=0x7fd7ec004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.985+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec041070 con 0x7fd7fc0650a0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.985+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7fc10d1c0 con 0x7fd7fc0650a0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.985+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd7ec009690 con 0x7fd7fc0650a0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.985+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd7ec03d8c0 con 0x7fd7fc0650a0 2024-06-15T00:54:09.987 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.985+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7fc10d680 con 0x7fd7fc0650a0 2024-06-15T00:54:09.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.986+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7fc0fda50 con 0x7fd7fc0650a0 2024-06-15T00:54:09.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.986+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd7ec02f030 con 0x7fd7fc0650a0 2024-06-15T00:54:09.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.986+0000 7fd7fa7fc640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7e4065430 0x7fd7e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:09.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.987+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd7ec0a3170 con 0x7fd7fc0650a0 2024-06-15T00:54:09.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.987+0000 7fd8014d0640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7e4065430 0x7fd7e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:09.990 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.987+0000 7fd8014d0640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7e4065430 0x7fd7e40678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0006fd0 tx=0x7fd7f0008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:09.991 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:09.991+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd7ec071620 con 0x7fd7fc0650a0 2024-06-15T00:54:09.991 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:09 smithi105 ceph-mon[28877]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:09.991 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:09 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:10.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:09 smithi143 ceph-mon[35860]: pgmap v171: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:10.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:09 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:10.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.191+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd7fc04b4b0 con 0x7fd7fc0650a0 2024-06-15T00:54:10.194 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.192+0000 7fd7fa7fc640 1 -- 172.21.15.105:0/2505907617 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd7ec074c40 con 0x7fd7fc0650a0 2024-06-15T00:54:10.195 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:10.195 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:10.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7e4065430 msgr2=0x7fd7e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:10.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7e4065430 0x7fd7e40678e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 crypto rx=0x7fd7f0006fd0 tx=0x7fd7f0008040 comp rx=0 tx=0).stop 2024-06-15T00:54:10.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 msgr2=0x7fd7fc112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:10.196 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc112fc0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 crypto rx=0x7fd7ec003400 tx=0x7fd7ec004d10 comp rx=0 tx=0).stop 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 shutdown_connections 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd7e4065430 0x7fd7e40678e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd7fc0fdeb0 0x7fd7fc113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 --2- 172.21.15.105:0/2505907617 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd7fc0650a0 0x7fd7fc112fc0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 >> 172.21.15.105:0/2505907617 conn(0x7fd7fc072110 msgr2=0x7fd7fc0f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 shutdown_connections 2024-06-15T00:54:10.197 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:10.194+0000 7fd803f35640 1 -- 172.21.15.105:0/2505907617 wait complete. 2024-06-15T00:54:11.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:10 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2505907617' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:11.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:10 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2505907617' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:12.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:11 smithi143 ceph-mon[35860]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:12.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:11 smithi105 ceph-mon[28877]: pgmap v172: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:13.872 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:14.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:13 smithi143 ceph-mon[35860]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:14.137 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:14.260 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:13 smithi105 ceph-mon[28877]: pgmap v173: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 -- 172.21.15.105:0/3612554731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2541011b0 msgr2=0x7fd254101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 --2- 172.21.15.105:0/3612554731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2541011b0 0x7fd254101580 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 crypto rx=0x7fd24c0048d0 tx=0x7fd24c009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 -- 172.21.15.105:0/3612554731 shutdown_connections 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 --2- 172.21.15.105:0/3612554731 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd254101b50 0x7fd2540fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 --2- 172.21.15.105:0/3612554731 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd2541011b0 0x7fd254101580 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 -- 172.21.15.105:0/3612554731 >> 172.21.15.105:0/3612554731 conn(0x7fd2540f6d60 msgr2=0x7fd2540f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 -- 172.21.15.105:0/3612554731 shutdown_connections 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.714+0000 7fd259414640 1 -- 172.21.15.105:0/3612554731 wait complete. 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd259414640 1 Processor -- start 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd259414640 1 -- start start 2024-06-15T00:54:15.716 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd259414640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2541011b0 0x7fd254112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd259414640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 0x7fd254113480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd259414640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd254113b10 con 0x7fd254101b50 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd259414640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd25410cf00 con 0x7fd2541011b0 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd2527fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 0x7fd254113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd2527fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 0x7fd254113480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52962/0 (socket says 172.21.15.105:52962) 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd2527fc640 1 -- 172.21.15.105:0/279960479 learned_addr learned my addr 172.21.15.105:0/279960479 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.715+0000 7fd252ffd640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2541011b0 0x7fd254112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd2527fc640 1 -- 172.21.15.105:0/279960479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2541011b0 msgr2=0x7fd254112f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd2527fc640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2541011b0 0x7fd254112f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.717 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd2527fc640 1 -- 172.21.15.105:0/279960479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd24c005040 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd2527fc640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 0x7fd254113480 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd24800bbb0 tx=0x7fd248009840 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd24800c940 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2480092e0 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd25410d1e0 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fd248007640 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.716+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd25410d700 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.717+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2540fc920 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.718+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fd248012070 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.718+0000 7fd23b7fe640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd23c065430 0x7fd23c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.718+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fd248082350 con 0x7fd254101b50 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.718+0000 7fd252ffd640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd23c065430 0x7fd23c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:15.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.718+0000 7fd252ffd640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd23c065430 0x7fd23c0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd24c000c00 tx=0x7fd24c035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:15.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.722+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fd248050880 con 0x7fd254101b50 2024-06-15T00:54:15.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.921+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fd2540fa9a0 con 0x7fd254101b50 2024-06-15T00:54:15.921 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.921+0000 7fd23b7fe640 1 -- 172.21.15.105:0/279960479 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fd248053ea0 con 0x7fd254101b50 2024-06-15T00:54:15.922 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:15.922 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:15.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd23c065430 msgr2=0x7fd23c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:15.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd23c065430 0x7fd23c0678e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 crypto rx=0x7fd24c000c00 tx=0x7fd24c035040 comp rx=0 tx=0).stop 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 msgr2=0x7fd254113480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 0x7fd254113480 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 crypto rx=0x7fd24800bbb0 tx=0x7fd248009840 comp rx=0 tx=0).stop 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 shutdown_connections 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fd23c065430 0x7fd23c0678e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fd254101b50 0x7fd254113480 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 --2- 172.21.15.105:0/279960479 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fd2541011b0 0x7fd254112f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 >> 172.21.15.105:0/279960479 conn(0x7fd2540f6d60 msgr2=0x7fd2540657f0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 shutdown_connections 2024-06-15T00:54:15.924 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:15.923+0000 7fd259414640 1 -- 172.21.15.105:0/279960479 wait complete. 2024-06-15T00:54:16.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:15 smithi143 ceph-mon[35860]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:16.265 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:15 smithi105 ceph-mon[28877]: pgmap v174: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:17.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:16 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/279960479' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:17.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:16 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/279960479' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:17.906 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:17 smithi105 ceph-mon[28877]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:18.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:17 smithi143 ceph-mon[35860]: pgmap v175: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:19.635 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:19.897 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:20.019 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:19 smithi105 ceph-mon[28877]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:20.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:19 smithi143 ceph-mon[35860]: pgmap v176: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:21.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 -- 172.21.15.105:0/55707917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc0102370 msgr2=0x7f8bc0102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:21.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 --2- 172.21.15.105:0/55707917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc0102370 0x7f8bc0102740 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00048d0 tx=0x7f8bb0009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 -- 172.21.15.105:0/55707917 shutdown_connections 2024-06-15T00:54:21.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 --2- 172.21.15.105:0/55707917 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fae50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.586 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 --2- 172.21.15.105:0/55707917 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc0102370 0x7f8bc0102740 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 -- 172.21.15.105:0/55707917 >> 172.21.15.105:0/55707917 conn(0x7f8bc0070a40 msgr2=0x7f8bc0070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 -- 172.21.15.105:0/55707917 shutdown_connections 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 -- 172.21.15.105:0/55707917 wait complete. 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.579+0000 7f8bc675b640 1 Processor -- start 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 -- start start 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fbbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bc0102370 0x7f8bc00fc100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc00fc640 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc00fc7b0 con 0x7f8bc0102370 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fbbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fbbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52982/0 (socket says 172.21.15.105:52982) 2024-06-15T00:54:21.587 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 -- 172.21.15.105:0/1708479064 learned_addr learned my addr 172.21.15.105:0/1708479064 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 -- 172.21.15.105:0/1708479064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bc0102370 msgr2=0x7f8bc00fc100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bc0102370 0x7f8bc00fc100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 -- 172.21.15.105:0/1708479064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bb0005040 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbffff640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fbbc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00048a0 tx=0x7f8bb0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb0038070 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8bb0032d30 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bc00fc9e0 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8bb0009680 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.580+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc01195b0 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.582+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc0065530 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.582+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8bb0030020 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.582+0000 7f8bbcff9640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8bac065430 0x7f8bac0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:21.588 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.582+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8bb00a2300 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.582+0000 7f8bbf7fe640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8bac065430 0x7f8bac0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:21.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.583+0000 7f8bbf7fe640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8bac065430 0x7f8bac0678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00fd500 tx=0x7f8ba8007cb0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:21.589 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.586+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8bb00707b0 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.784+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8bc004b480 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.785 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.785+0000 7f8bbcff9640 1 -- 172.21.15.105:0/1708479064 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8bb0073dd0 con 0x7f8bc00fa9e0 2024-06-15T00:54:21.785 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:21.785 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8bac065430 msgr2=0x7f8bac0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8bac065430 0x7f8bac0678e0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 crypto rx=0x7f8bc00fd500 tx=0x7f8ba8007cb0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 msgr2=0x7f8bc00fbbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fbbc0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 crypto rx=0x7f8bb00048a0 tx=0x7f8bb0009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 shutdown_connections 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8bac065430 0x7f8bac0678e0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.786+0000 7f8bc675b640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8bc0102370 0x7f8bc00fc100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.787+0000 7f8bc675b640 1 --2- 172.21.15.105:0/1708479064 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8bc00fa9e0 0x7f8bc00fbbc0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:21.787 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.787+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 >> 172.21.15.105:0/1708479064 conn(0x7f8bc0070a40 msgr2=0x7f8bc00f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:21.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.787+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 shutdown_connections 2024-06-15T00:54:21.788 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:21.787+0000 7f8bc675b640 1 -- 172.21.15.105:0/1708479064 wait complete. 2024-06-15T00:54:22.080 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:21 smithi105 ceph-mon[28877]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:22.081 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:21 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1708479064' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:22.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:21 smithi143 ceph-mon[35860]: pgmap v177: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:22.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:21 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1708479064' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:24.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:23 smithi143 ceph-mon[35860]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:23 smithi105 ceph-mon[28877]: pgmap v178: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:25.137 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:24 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:25.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:25.415 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:25.681 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:26.101 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:25 smithi105 ceph-mon[28877]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:26.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:25 smithi143 ceph-mon[35860]: pgmap v179: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:27.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/3207871680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 msgr2=0x7f4d68101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:27.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/3207871680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68101590 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 crypto rx=0x7f4d580048d0 tx=0x7f4d58009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/3207871680 shutdown_connections 2024-06-15T00:54:27.334 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/3207871680 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d68101b60 0x7f4d680fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/3207871680 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68101590 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/3207871680 >> 172.21.15.105:0/3207871680 conn(0x7f4d680f6d70 msgr2=0x7f4d680f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/3207871680 shutdown_connections 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/3207871680 wait complete. 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 Processor -- start 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.317+0000 7f4d6ddc9640 1 -- start start 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d6ddc9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d6ddc9640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d68101b60 0x7f4d681134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d6ddc9640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d68113b50 con 0x7f4d681011c0 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d6ddc9640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d6810cf00 con 0x7f4d68101b60 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:27.335 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68112f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34528/0 (socket says 172.21.15.105:34528) 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 -- 172.21.15.105:0/1751298444 learned_addr learned my addr 172.21.15.105:0/1751298444 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d5ffff640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d68101b60 0x7f4d681134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 -- 172.21.15.105:0/1751298444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d68101b60 msgr2=0x7f4d681134c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d68101b60 0x7f4d681134c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 -- 172.21.15.105:0/1751298444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d58005040 con 0x7f4d681011c0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d677fe640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68112f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4d580048a0 tx=0x7f4d58009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d58038070 con 0x7f4d681011c0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d6810d180 con 0x7f4d681011c0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d58032d30 con 0x7f4d681011c0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4d58009680 con 0x7f4d681011c0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.318+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d6810d640 con 0x7f4d681011c0 2024-06-15T00:54:27.336 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.319+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d680fc930 con 0x7f4d681011c0 2024-06-15T00:54:27.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.320+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4d58030020 con 0x7f4d681011c0 2024-06-15T00:54:27.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.320+0000 7f4d64ff9640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4d54065430 0x7f4d540678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:27.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.320+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4d580a2300 con 0x7f4d681011c0 2024-06-15T00:54:27.337 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.320+0000 7f4d5ffff640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4d54065430 0x7f4d540678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:27.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.320+0000 7f4d5ffff640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4d54065430 0x7f4d540678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006fd0 tx=0x7f4d50008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:27.338 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.324+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4d580707b0 con 0x7f4d681011c0 2024-06-15T00:54:27.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.521+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4d6804b4b0 con 0x7f4d681011c0 2024-06-15T00:54:27.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.521+0000 7f4d64ff9640 1 -- 172.21.15.105:0/1751298444 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4d58073dd0 con 0x7f4d681011c0 2024-06-15T00:54:27.522 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:27.522 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:27.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4d54065430 msgr2=0x7f4d540678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4d54065430 0x7f4d540678e0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 crypto rx=0x7f4d50006fd0 tx=0x7f4d50008040 comp rx=0 tx=0).stop 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 msgr2=0x7f4d68112f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68112f80 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 crypto rx=0x7f4d580048a0 tx=0x7f4d58009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 shutdown_connections 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4d54065430 0x7f4d540678e0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4d68101b60 0x7f4d681134c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 --2- 172.21.15.105:0/1751298444 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4d681011c0 0x7f4d68112f80 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 >> 172.21.15.105:0/1751298444 conn(0x7f4d680f6d70 msgr2=0x7f4d680f8570 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 shutdown_connections 2024-06-15T00:54:27.524 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:27.523+0000 7f4d6ddc9640 1 -- 172.21.15.105:0/1751298444 wait complete. 2024-06-15T00:54:28.120 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:27 smithi105 ceph-mon[28877]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:28.120 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:27 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1751298444' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:28.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:27 smithi143 ceph-mon[35860]: pgmap v180: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:28.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:27 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1751298444' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:30.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:29 smithi143 ceph-mon[35860]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:29 smithi105 ceph-mon[28877]: pgmap v181: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:31.144 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:31.414 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:32.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:31 smithi143 ceph-mon[35860]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:32.202 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:31 smithi105 ceph-mon[28877]: pgmap v182: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:33.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 -- 172.21.15.105:0/1728575748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 msgr2=0x7f57900fb680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:33.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 --2- 172.21.15.105:0/1728575748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f57900fb680 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 crypto rx=0x7f578c0048d0 tx=0x7f578c009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 -- 172.21.15.105:0/1728575748 shutdown_connections 2024-06-15T00:54:33.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 --2- 172.21.15.105:0/1728575748 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 0x7f57901000d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 --2- 172.21.15.105:0/1728575748 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f57900fb680 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 -- 172.21.15.105:0/1728575748 >> 172.21.15.105:0/1728575748 conn(0x7f57900f6d90 msgr2=0x7f57900f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 -- 172.21.15.105:0/1728575748 shutdown_connections 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.049+0000 7f5797a75640 1 -- 172.21.15.105:0/1728575748 wait complete. 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5797a75640 1 Processor -- start 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5797a75640 1 -- start start 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5797a75640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f5790119680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5797a75640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 0x7f5790119bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5797a75640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f579011a250 con 0x7f57900fb2b0 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5797a75640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5790113690 con 0x7f57900fbc50 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f5790119680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:33.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f5790119680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51928/0 (socket says 172.21.15.105:51928) 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 -- 172.21.15.105:0/489699187 learned_addr learned my addr 172.21.15.105:0/489699187 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795010640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 0x7f5790119bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 -- 172.21.15.105:0/489699187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 msgr2=0x7f5790119bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 0x7f5790119bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 -- 172.21.15.105:0/489699187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f578c005040 con 0x7f57900fb2b0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795010640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 0x7f5790119bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.050+0000 7f5795811640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f5790119680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f578c0048a0 tx=0x7f578c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.051+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f578c038070 con 0x7f57900fb2b0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.051+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f578c032d30 con 0x7f57900fb2b0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.051+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5790113910 con 0x7f57900fb2b0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.051+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f578c009680 con 0x7f57900fb2b0 2024-06-15T00:54:33.054 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.051+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5790113dd0 con 0x7f57900fb2b0 2024-06-15T00:54:33.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.052+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57900fa9f0 con 0x7f57900fb2b0 2024-06-15T00:54:33.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.052+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f578c030020 con 0x7f57900fb2b0 2024-06-15T00:54:33.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.053+0000 7f57867fc640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f577c065430 0x7f577c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:33.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.053+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f578c0a21f0 con 0x7f57900fb2b0 2024-06-15T00:54:33.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.053+0000 7f5795010640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f577c065430 0x7f577c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:33.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.053+0000 7f5795010640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f577c065430 0x7f577c0678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5780006fd0 tx=0x7f5780008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:33.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.057+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f578c070720 con 0x7f57900fb2b0 2024-06-15T00:54:33.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.255+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f5790114dc0 con 0x7f57900fb2b0 2024-06-15T00:54:33.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.255+0000 7f57867fc640 1 -- 172.21.15.105:0/489699187 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f578c073d40 con 0x7f57900fb2b0 2024-06-15T00:54:33.256 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:33.256 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:33.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f577c065430 msgr2=0x7f577c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:33.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f577c065430 0x7f577c0678e0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 crypto rx=0x7f5780006fd0 tx=0x7f5780008040 comp rx=0 tx=0).stop 2024-06-15T00:54:33.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 msgr2=0x7f5790119680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:33.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f5790119680 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 crypto rx=0x7f578c0048a0 tx=0x7f578c009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 shutdown_connections 2024-06-15T00:54:33.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f577c065430 0x7f577c0678e0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f57900fbc50 0x7f5790119bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 --2- 172.21.15.105:0/489699187 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f57900fb2b0 0x7f5790119680 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:33.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 >> 172.21.15.105:0/489699187 conn(0x7f57900f6d90 msgr2=0x7f57900f8420 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:33.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 shutdown_connections 2024-06-15T00:54:33.260 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:33.257+0000 7f5797a75640 1 -- 172.21.15.105:0/489699187 wait complete. 2024-06-15T00:54:34.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:33 smithi143 ceph-mon[35860]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:34.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:33 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/489699187' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:33 smithi105 ceph-mon[28877]: pgmap v183: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:33 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/489699187' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:36.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:35 smithi143 ceph-mon[35860]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:36.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:35 smithi105 ceph-mon[28877]: pgmap v184: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:36.842 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:37.111 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:38.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:37 smithi143 ceph-mon[35860]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:38.187 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:37 smithi105 ceph-mon[28877]: pgmap v185: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 -- 172.21.15.105:0/4083156759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fb570 msgr2=0x7f26300fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 --2- 172.21.15.105:0/4083156759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fb570 0x7f26300fb940 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 crypto rx=0x7f262c0048d0 tx=0x7f262c009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 -- 172.21.15.105:0/4083156759 shutdown_connections 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 --2- 172.21.15.105:0/4083156759 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fbe80 0x7f26300fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 --2- 172.21.15.105:0/4083156759 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fb570 0x7f26300fb940 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 -- 172.21.15.105:0/4083156759 >> 172.21.15.105:0/4083156759 conn(0x7f2630072110 msgr2=0x7f2630072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 -- 172.21.15.105:0/4083156759 shutdown_connections 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 -- 172.21.15.105:0/4083156759 wait complete. 2024-06-15T00:54:38.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 Processor -- start 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.712+0000 7f2636cf5640 1 -- start start 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2636cf5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fb570 0x7f2630112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2636cf5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 0x7f2630113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2636cf5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2630113b90 con 0x7f26300fbe80 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2636cf5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f263010cf80 con 0x7f26300fb570 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 0x7f2630113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 0x7f2630113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:51944/0 (socket says 172.21.15.105:51944) 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 -- 172.21.15.105:0/2507800024 learned_addr learned my addr 172.21.15.105:0/2507800024 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2634a91640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fb570 0x7f2630112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 -- 172.21.15.105:0/2507800024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fb570 msgr2=0x7f2630112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fb570 0x7f2630112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 -- 172.21.15.105:0/2507800024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c005040 con 0x7f26300fbe80 2024-06-15T00:54:38.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2634a91640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fb570 0x7f2630112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2627fff640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 0x7f2630113500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f262000b790 tx=0x7f26200091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f262000c700 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f263010d260 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f262000cd10 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2620013370 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.713+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f263010d780 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.714+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26300fc920 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.715+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2620012070 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.715+0000 7f26257fa640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2618065430 0x7f26180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.715+0000 7f2634a91640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2618065430 0x7f26180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.715+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2620082170 con 0x7f26300fbe80 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.716+0000 7f2634a91640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2618065430 0x7f26180678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f262c003080 tx=0x7f262c009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:38.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.719+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2620050620 con 0x7f26300fbe80 2024-06-15T00:54:38.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.919+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f263010e670 con 0x7f26300fbe80 2024-06-15T00:54:38.920 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.919+0000 7f26257fa640 1 -- 172.21.15.105:0/2507800024 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2620053c40 con 0x7f26300fbe80 2024-06-15T00:54:38.920 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:38.920 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.921+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2618065430 msgr2=0x7f26180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.921+0000 7f2636cf5640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2618065430 0x7f26180678e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 crypto rx=0x7f262c003080 tx=0x7f262c009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.921+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 msgr2=0x7f2630113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.921+0000 7f2636cf5640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 0x7f2630113500 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 crypto rx=0x7f262000b790 tx=0x7f26200091d0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 shutdown_connections 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2618065430 0x7f26180678e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.922 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f26300fbe80 0x7f2630113500 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 --2- 172.21.15.105:0/2507800024 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f26300fb570 0x7f2630112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:38.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 >> 172.21.15.105:0/2507800024 conn(0x7f2630072110 msgr2=0x7f26301000a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:38.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 shutdown_connections 2024-06-15T00:54:38.923 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:38.922+0000 7f2636cf5640 1 -- 172.21.15.105:0/2507800024 wait complete. 2024-06-15T00:54:40.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:39 smithi143 ceph-mon[35860]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:40.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:39 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2507800024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:40.136 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:39 smithi105 ceph-mon[28877]: pgmap v186: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:39 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2507800024' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:42.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:41 smithi143 ceph-mon[35860]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:42.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:41 smithi105 ceph-mon[28877]: pgmap v187: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:42.615 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:42.880 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:44.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:43 smithi143 ceph-mon[35860]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:44.185 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:43 smithi105 ceph-mon[28877]: pgmap v188: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.559+0000 7fdb4cc09640 1 -- 172.21.15.105:0/835117959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb481011b0 msgr2=0x7fdb48101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.559+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/835117959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb481011b0 0x7fdb48101580 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009c50 tx=0x7fdb40004d10 comp rx=0 tx=0).stop 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- 172.21.15.105:0/835117959 shutdown_connections 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/835117959 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb48101b50 0x7fdb480fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/835117959 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb481011b0 0x7fdb48101580 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- 172.21.15.105:0/835117959 >> 172.21.15.105:0/835117959 conn(0x7fdb480f6d80 msgr2=0x7fdb480f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- 172.21.15.105:0/835117959 shutdown_connections 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- 172.21.15.105:0/835117959 wait complete. 2024-06-15T00:54:44.562 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 Processor -- start 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- start start 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb481011b0 0x7fdb48112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 0x7fdb481134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb48113b80 con 0x7fdb48101b50 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb4cc09640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb4810cf30 con 0x7fdb481011b0 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb45d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 0x7fdb481134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb45d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 0x7fdb481134f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34876/0 (socket says 172.21.15.105:34876) 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.560+0000 7fdb45d9b640 1 -- 172.21.15.105:0/3705371860 learned_addr learned my addr 172.21.15.105:0/3705371860 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb4659c640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb481011b0 0x7fdb48112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb45d9b640 1 -- 172.21.15.105:0/3705371860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb481011b0 msgr2=0x7fdb48112fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb45d9b640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb481011b0 0x7fdb48112fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb45d9b640 1 -- 172.21.15.105:0/3705371860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb400098b0 con 0x7fdb48101b50 2024-06-15T00:54:44.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb45d9b640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 0x7fdb481134f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000b790 tx=0x7fdb300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:44.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb3000c700 con 0x7fdb48101b50 2024-06-15T00:54:44.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb3000cd10 con 0x7fdb48101b50 2024-06-15T00:54:44.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb4810d210 con 0x7fdb48101b50 2024-06-15T00:54:44.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdb30013370 con 0x7fdb48101b50 2024-06-15T00:54:44.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.561+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb4810d730 con 0x7fdb48101b50 2024-06-15T00:54:44.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.562+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb480fc920 con 0x7fdb48101b50 2024-06-15T00:54:44.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.563+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdb30012070 con 0x7fdb48101b50 2024-06-15T00:54:44.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.563+0000 7fdb3effd640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdb34065430 0x7fdb340678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:44.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.563+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdb30082030 con 0x7fdb48101b50 2024-06-15T00:54:44.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.563+0000 7fdb4659c640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdb34065430 0x7fdb340678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:44.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.563+0000 7fdb4659c640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdb34065430 0x7fdb340678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009c20 tx=0x7fdb40004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:44.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.567+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdb30050560 con 0x7fdb48101b50 2024-06-15T00:54:44.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.763+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdb48108260 con 0x7fdb48101b50 2024-06-15T00:54:44.764 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.764+0000 7fdb3effd640 1 -- 172.21.15.105:0/3705371860 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdb30053b80 con 0x7fdb48101b50 2024-06-15T00:54:44.764 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:44.764 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:44.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdb34065430 msgr2=0x7fdb340678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:44.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdb34065430 0x7fdb340678e0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 crypto rx=0x7fdb40009c20 tx=0x7fdb40004d10 comp rx=0 tx=0).stop 2024-06-15T00:54:44.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 msgr2=0x7fdb481134f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 0x7fdb481134f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 crypto rx=0x7fdb3000b790 tx=0x7fdb300091d0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 shutdown_connections 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdb34065430 0x7fdb340678e0 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdb48101b50 0x7fdb481134f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 --2- 172.21.15.105:0/3705371860 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdb481011b0 0x7fdb48112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 >> 172.21.15.105:0/3705371860 conn(0x7fdb480f6d80 msgr2=0x7fdb480f8500 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 shutdown_connections 2024-06-15T00:54:44.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:44.766+0000 7fdb4cc09640 1 -- 172.21.15.105:0/3705371860 wait complete. 2024-06-15T00:54:45.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:44 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3705371860' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:45.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:44 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3705371860' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:45 smithi105 ceph-mon[28877]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:45 smithi143 ceph-mon[35860]: pgmap v189: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:48.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:47 smithi105 ceph-mon[28877]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:48.337 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:48.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:47 smithi143 ceph-mon[35860]: pgmap v190: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:48.601 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:49.971 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:49 smithi105 ceph-mon[28877]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:50.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 -- 172.21.15.105:0/313721894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 msgr2=0x7f2cfc0fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 --2- 172.21.15.105:0/313721894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc0fcd80 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40048d0 tx=0x7f2ce4009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 -- 172.21.15.105:0/313721894 shutdown_connections 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 --2- 172.21.15.105:0/313721894 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc0fcd80 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 --2- 172.21.15.105:0/313721894 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 0x7f2cfc0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 -- 172.21.15.105:0/313721894 >> 172.21.15.105:0/313721894 conn(0x7f2cfc072110 msgr2=0x7f2cfc072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 -- 172.21.15.105:0/313721894 shutdown_connections 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 -- 172.21.15.105:0/313721894 wait complete. 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 Processor -- start 2024-06-15T00:54:50.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.181+0000 7f2d012a1640 1 -- start start 2024-06-15T00:54:50.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2d012a1640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 0x7f2cfc111ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2d012a1640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc1123e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2d012a1640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc112a70 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2d012a1640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2cfc10be20 con 0x7f2cfc0fb570 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc1123e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc1123e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:34898/0 (socket says 172.21.15.105:34898) 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 -- 172.21.15.105:0/3404813769 learned_addr learned my addr 172.21.15.105:0/3404813769 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfaffd640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 0x7f2cfc111ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 -- 172.21.15.105:0/3404813769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 msgr2=0x7f2cfc111ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 0x7f2cfc111ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 -- 172.21.15.105:0/3404813769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ce4005040 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfaffd640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 0x7f2cfc111ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cfa7fc640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc1123e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40048a0 tx=0x7f2ce4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:50.185 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ce403a070 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2ce4008ed0 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2cfc10c0a0 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.182+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2ce40338c0 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.183+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2cfc10c560 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.183+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2cfc0fc920 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.184+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2ce4030020 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.184+0000 7f2cf37fe640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2ce8065430 0x7f2ce80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.184+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f2ce40a44d0 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.184+0000 7f2cfaffd640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2ce8065430 0x7f2ce80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:50.186 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.185+0000 7f2cfaffd640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2ce8065430 0x7f2ce80678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4007fd0 tx=0x7f2cf40074f0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:50.188 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.188+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2ce4072a00 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:49 smithi143 ceph-mon[35860]: pgmap v191: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.385+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f2cfc072c20 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.386+0000 7f2cf37fe640 1 -- 172.21.15.105:0/3404813769 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2ce4076020 con 0x7f2cfc0fbe80 2024-06-15T00:54:50.386 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:50.387 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:50.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2ce8065430 msgr2=0x7f2ce80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:50.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2ce8065430 0x7f2ce80678e0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 crypto rx=0x7f2cf4007fd0 tx=0x7f2cf40074f0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 msgr2=0x7f2cfc1123e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:50.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc1123e0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 crypto rx=0x7f2ce40048a0 tx=0x7f2ce4009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 shutdown_connections 2024-06-15T00:54:50.388 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2ce8065430 0x7f2ce80678e0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f2cfc0fbe80 0x7f2cfc1123e0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 --2- 172.21.15.105:0/3404813769 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2cfc0fb570 0x7f2cfc111ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:50.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 >> 172.21.15.105:0/3404813769 conn(0x7f2cfc072110 msgr2=0x7f2cfc0f8710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:50.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 shutdown_connections 2024-06-15T00:54:50.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:50.388+0000 7f2d012a1640 1 -- 172.21.15.105:0/3404813769 wait complete. 2024-06-15T00:54:51.001 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:50 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3404813769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:51.001 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:50 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:54:51.378 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:50 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3404813769' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:51.378 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:50 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:54:52.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:51 smithi105 ceph-mon[28877]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:52.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:54:52.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:54:52.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:54:52.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:51 smithi143 ceph-mon[35860]: pgmap v192: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:52.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:54:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:54:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:54:54.025 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:54:54.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:53 smithi105 ceph-mon[28877]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:54.287 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:54:54.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:53 smithi143 ceph-mon[35860]: pgmap v193: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:55.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:54 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:55.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:54 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- 172.21.15.105:0/2622096246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42741022a0 msgr2=0x7f4274102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 --2- 172.21.15.105:0/2622096246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42741022a0 0x7f4274102670 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 crypto rx=0x7f42640048d0 tx=0x7f4264009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- 172.21.15.105:0/2622096246 shutdown_connections 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 --2- 172.21.15.105:0/2622096246 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4274102c40 0x7f42740fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 --2- 172.21.15.105:0/2622096246 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f42741022a0 0x7f4274102670 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- 172.21.15.105:0/2622096246 >> 172.21.15.105:0/2622096246 conn(0x7f42740f7e50 msgr2=0x7f42740fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- 172.21.15.105:0/2622096246 shutdown_connections 2024-06-15T00:54:55.983 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- 172.21.15.105:0/2622096246 wait complete. 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 Processor -- start 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- start start 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42741022a0 0x7f4274112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 0x7f4274113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4274113b90 con 0x7f4274102c40 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f4279287640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f427410cf40 con 0x7f42741022a0 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.962+0000 7f426bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 0x7f4274113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 0x7f4274113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42452/0 (socket says 172.21.15.105:42452) 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426bfff640 1 -- 172.21.15.105:0/1032658333 learned_addr learned my addr 172.21.15.105:0/1032658333 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f4272ffd640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42741022a0 0x7f4274112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426bfff640 1 -- 172.21.15.105:0/1032658333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42741022a0 msgr2=0x7f4274112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426bfff640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42741022a0 0x7f4274112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426bfff640 1 -- 172.21.15.105:0/1032658333 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4264005040 con 0x7f4274102c40 2024-06-15T00:54:55.984 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f4272ffd640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42741022a0 0x7f4274112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426bfff640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 0x7f4274113500 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f425c00b790 tx=0x7f425c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425c00c700 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f427410d220 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f425c00cd10 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f425c015e70 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.963+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f427410d740 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.964+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f42740fda10 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.965+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f425c012070 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.965+0000 7f426b7fe640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4260065430 0x7f42600678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.965+0000 7f4272ffd640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4260065430 0x7f42600678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.965+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f425c081dc0 con 0x7f4274102c40 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.966+0000 7f4272ffd640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4260065430 0x7f42600678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f426400b020 tx=0x7f4264009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:54:55.985 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:55.969+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f425c050270 con 0x7f4274102c40 2024-06-15T00:54:56.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:55 smithi105 ceph-mon[28877]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:56.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.167+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4274109380 con 0x7f4274102c40 2024-06-15T00:54:56.168 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.168+0000 7f426b7fe640 1 -- 172.21.15.105:0/1032658333 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f425c053890 con 0x7f4274102c40 2024-06-15T00:54:56.168 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:54:56.168 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4260065430 msgr2=0x7f42600678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4260065430 0x7f42600678e0 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 crypto rx=0x7f426400b020 tx=0x7f4264009da0 comp rx=0 tx=0).stop 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 msgr2=0x7f4274113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 0x7f4274113500 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 crypto rx=0x7f425c00b790 tx=0x7f425c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 shutdown_connections 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4260065430 0x7f42600678e0 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4274102c40 0x7f4274113500 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 --2- 172.21.15.105:0/1032658333 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f42741022a0 0x7f4274112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.169+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 >> 172.21.15.105:0/1032658333 conn(0x7f42740f7e50 msgr2=0x7f42740f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.170+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 shutdown_connections 2024-06-15T00:54:56.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:54:56.170+0000 7f4279287640 1 -- 172.21.15.105:0/1032658333 wait complete. 2024-06-15T00:54:56.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:55 smithi143 ceph-mon[35860]: pgmap v194: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:56 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1032658333' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:57.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:56 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1032658333' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:54:58.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:57 smithi105 ceph-mon[28877]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:58.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:57 smithi143 ceph-mon[35860]: pgmap v195: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:54:59.896 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:00.163 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:00.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:54:59 smithi105 ceph-mon[28877]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:00.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:54:59 smithi143 ceph-mon[35860]: pgmap v196: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2997300114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac1011d0 msgr2=0x7fb8ac1015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2997300114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac1015a0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 crypto rx=0x7fb8a00048d0 tx=0x7fb8a0009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2997300114 shutdown_connections 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2997300114 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac101b70 0x7fb8ac0fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2997300114 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac1015a0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2997300114 >> 172.21.15.105:0/2997300114 conn(0x7fb8ac0f6dc0 msgr2=0x7fb8ac0f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2997300114 shutdown_connections 2024-06-15T00:55:01.898 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.897+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2997300114 wait complete. 2024-06-15T00:55:01.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8b2d5b640 1 Processor -- start 2024-06-15T00:55:01.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8b2d5b640 1 -- start start 2024-06-15T00:55:01.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8b2d5b640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac112f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:01.900 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8b2d5b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 0x7fb8ac113440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8b2d5b640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac113a40 con 0x7fb8ac101b70 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8b2d5b640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8ac10ce80 con 0x7fb8ac1011d0 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.898+0000 7fb8abfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 0x7fb8ac113440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8abfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 0x7fb8ac113440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:42468/0 (socket says 172.21.15.105:42468) 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8abfff640 1 -- 172.21.15.105:0/2238797407 learned_addr learned my addr 172.21.15.105:0/2238797407 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8b0af7640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac112f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8abfff640 1 -- 172.21.15.105:0/2238797407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac1011d0 msgr2=0x7fb8ac112f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8abfff640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac112f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8abfff640 1 -- 172.21.15.105:0/2238797407 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8a0005040 con 0x7fb8ac101b70 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8b0af7640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8abfff640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 0x7fb8ac113440 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c830 tx=0x7fb898004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb89800a850 con 0x7fb8ac101b70 2024-06-15T00:55:01.901 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb89800ae60 con 0x7fb8ac101b70 2024-06-15T00:55:01.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8ac10d160 con 0x7fb8ac101b70 2024-06-15T00:55:01.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb8980113a0 con 0x7fb8ac101b70 2024-06-15T00:55:01.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.899+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8ac10d700 con 0x7fb8ac101b70 2024-06-15T00:55:01.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.900+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8ac0fc940 con 0x7fb8ac101b70 2024-06-15T00:55:01.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.901+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb898012400 con 0x7fb8ac101b70 2024-06-15T00:55:01.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.901+0000 7fb8a97fa640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb89c065430 0x7fb89c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:01.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.901+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb8980826e0 con 0x7fb8ac101b70 2024-06-15T00:55:01.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.901+0000 7fb8b0af7640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb89c065430 0x7fb89c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:01.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.902+0000 7fb8b0af7640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb89c065430 0x7fb89c0678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007ac0 tx=0x7fb8a0035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:01.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:01.905+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb898050cd0 con 0x7fb8ac101b70 2024-06-15T00:55:02.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:01 smithi105 ceph-mon[28877]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:02.106 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.105+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb8ac0fa9c0 con 0x7fb8ac101b70 2024-06-15T00:55:02.106 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.106+0000 7fb8a97fa640 1 -- 172.21.15.105:0/2238797407 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb8980542f0 con 0x7fb8ac101b70 2024-06-15T00:55:02.106 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:02.106 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:02.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb89c065430 msgr2=0x7fb89c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:02.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb89c065430 0x7fb89c0678e0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 crypto rx=0x7fb8a0007ac0 tx=0x7fb8a0035040 comp rx=0 tx=0).stop 2024-06-15T00:55:02.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 msgr2=0x7fb8ac113440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:02.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 0x7fb8ac113440 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 crypto rx=0x7fb89800c830 tx=0x7fb898004430 comp rx=0 tx=0).stop 2024-06-15T00:55:02.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 shutdown_connections 2024-06-15T00:55:02.108 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb89c065430 0x7fb89c0678e0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:02.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb8ac101b70 0x7fb8ac113440 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:02.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 --2- 172.21.15.105:0/2238797407 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb8ac1011d0 0x7fb8ac112f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:02.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 >> 172.21.15.105:0/2238797407 conn(0x7fb8ac0f6dc0 msgr2=0x7fb8ac065800 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:02.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 shutdown_connections 2024-06-15T00:55:02.109 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:02.108+0000 7fb8b2d5b640 1 -- 172.21.15.105:0/2238797407 wait complete. 2024-06-15T00:55:02.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:01 smithi143 ceph-mon[35860]: pgmap v197: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:03.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:02 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2238797407' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:03.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:02 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2238797407' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:04.135 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:03 smithi143 ceph-mon[35860]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:04.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:03 smithi105 ceph-mon[28877]: pgmap v198: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:05.812 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:06.081 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:06.204 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:05 smithi105 ceph-mon[28877]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:06.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:05 smithi143 ceph-mon[35860]: pgmap v199: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:07.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 -- 172.21.15.105:0/1125618437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 msgr2=0x7f983c1026c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:07.687 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 --2- 172.21.15.105:0/1125618437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c1026c0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 crypto rx=0x7f982c0048d0 tx=0x7f982c009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 -- 172.21.15.105:0/1125618437 shutdown_connections 2024-06-15T00:55:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 --2- 172.21.15.105:0/1125618437 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f983c102c90 0x7f983c0fa9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 --2- 172.21.15.105:0/1125618437 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c1026c0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 -- 172.21.15.105:0/1125618437 >> 172.21.15.105:0/1125618437 conn(0x7f983c070a40 msgr2=0x7f983c070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 -- 172.21.15.105:0/1125618437 shutdown_connections 2024-06-15T00:55:07.688 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.663+0000 7f9842524640 1 -- 172.21.15.105:0/1125618437 wait complete. 2024-06-15T00:55:07.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f9842524640 1 Processor -- start 2024-06-15T00:55:07.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f9842524640 1 -- start start 2024-06-15T00:55:07.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f9842524640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c112ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:07.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f9842524640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f983c102c90 0x7f983c113530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:07.689 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f9842524640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c113b30 con 0x7f983c1022f0 2024-06-15T00:55:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f9842524640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f983c10cf70 con 0x7f983c102c90 2024-06-15T00:55:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c112ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c112ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50302/0 (socket says 172.21.15.105:50302) 2024-06-15T00:55:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983bfff640 1 -- 172.21.15.105:0/530093601 learned_addr learned my addr 172.21.15.105:0/530093601 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983b7fe640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f983c102c90 0x7f983c113530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:07.690 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983bfff640 1 -- 172.21.15.105:0/530093601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f983c102c90 msgr2=0x7f983c113530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983bfff640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f983c102c90 0x7f983c113530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.664+0000 7f983bfff640 1 -- 172.21.15.105:0/530093601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f982c005040 con 0x7f983c1022f0 2024-06-15T00:55:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.665+0000 7f983bfff640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c112ff0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f982c0048a0 tx=0x7f982c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.665+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982c038070 con 0x7f983c1022f0 2024-06-15T00:55:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.665+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f983c10d1f0 con 0x7f983c1022f0 2024-06-15T00:55:07.691 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.665+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f982c02cef0 con 0x7f983c1022f0 2024-06-15T00:55:07.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.665+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f982c009580 con 0x7f983c1022f0 2024-06-15T00:55:07.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.665+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f983c10d6b0 con 0x7f983c1022f0 2024-06-15T00:55:07.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.666+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f983c065510 con 0x7f983c1022f0 2024-06-15T00:55:07.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.666+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f982c030020 con 0x7f983c1022f0 2024-06-15T00:55:07.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.666+0000 7f9838ff9640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9824065430 0x7f98240678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:07.692 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.667+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f982c0a2970 con 0x7f983c1022f0 2024-06-15T00:55:07.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.667+0000 7f983b7fe640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9824065430 0x7f98240678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:07.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.667+0000 7f983b7fe640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9824065430 0x7f98240678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9830007fd0 tx=0x7f9830009e90 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:07.693 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.671+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f982c070e20 con 0x7f983c1022f0 2024-06-15T00:55:07.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.868+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f983c04b480 con 0x7f983c1022f0 2024-06-15T00:55:07.869 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.869+0000 7f9838ff9640 1 -- 172.21.15.105:0/530093601 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f982c074440 con 0x7f983c1022f0 2024-06-15T00:55:07.869 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:07.869 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:07.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9824065430 msgr2=0x7f98240678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:07.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9824065430 0x7f98240678e0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 crypto rx=0x7f9830007fd0 tx=0x7f9830009e90 comp rx=0 tx=0).stop 2024-06-15T00:55:07.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 msgr2=0x7f983c112ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:07.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c112ff0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 crypto rx=0x7f982c0048a0 tx=0x7f982c009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.871 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 shutdown_connections 2024-06-15T00:55:07.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9824065430 0x7f98240678e0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f983c102c90 0x7f983c113530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 --2- 172.21.15.105:0/530093601 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f983c1022f0 0x7f983c112ff0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:07.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 >> 172.21.15.105:0/530093601 conn(0x7f983c070a40 msgr2=0x7f983c0f94e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:07.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 shutdown_connections 2024-06-15T00:55:07.872 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:07.871+0000 7f9842524640 1 -- 172.21.15.105:0/530093601 wait complete. 2024-06-15T00:55:08.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:07 smithi105 ceph-mon[28877]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:08.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:07 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/530093601' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:08.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:07 smithi143 ceph-mon[35860]: pgmap v200: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:08.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:07 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/530093601' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:10.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:09 smithi105 ceph-mon[28877]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:10.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:09 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:10.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:09 smithi143 ceph-mon[35860]: pgmap v201: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:10.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:09 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:11.569 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:11.825 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:12.194 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:11 smithi105 ceph-mon[28877]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:12.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:11 smithi143 ceph-mon[35860]: pgmap v202: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:13.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 -- 172.21.15.105:0/2996368836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fbe80 msgr2=0x7f66b00fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:13.593 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 --2- 172.21.15.105:0/2996368836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fbe80 0x7f66b00fcd80 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 crypto rx=0x7f66a40048d0 tx=0x7f66a4009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 -- 172.21.15.105:0/2996368836 shutdown_connections 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 --2- 172.21.15.105:0/2996368836 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fbe80 0x7f66b00fcd80 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 --2- 172.21.15.105:0/2996368836 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b00fb570 0x7f66b00fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 -- 172.21.15.105:0/2996368836 >> 172.21.15.105:0/2996368836 conn(0x7f66b0072110 msgr2=0x7f66b0072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 -- 172.21.15.105:0/2996368836 shutdown_connections 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.515+0000 7f66b6525640 1 -- 172.21.15.105:0/2996368836 wait complete. 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66b6525640 1 Processor -- start 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66b6525640 1 -- start start 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66b6525640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 0x7f66b010dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66b6525640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b00fbe80 0x7f66b010e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:13.594 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66b6525640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b010ea50 con 0x7f66b00fb570 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66b6525640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b010ebc0 con 0x7f66b00fbe80 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 0x7f66b010dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 0x7f66b010dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59752/0 (socket says 172.21.15.105:59752) 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 -- 172.21.15.105:0/873718426 learned_addr learned my addr 172.21.15.105:0/873718426 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66af7fe640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b00fbe80 0x7f66b010e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 -- 172.21.15.105:0/873718426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b00fbe80 msgr2=0x7f66b010e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b00fbe80 0x7f66b010e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 -- 172.21.15.105:0/873718426 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a4005040 con 0x7f66b00fb570 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.516+0000 7f66affff640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 0x7f66b010dec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f66a000c790 tx=0x7f66a00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.517+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a000a820 con 0x7f66b00fb570 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.517+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66a000ae30 con 0x7f66b00fb570 2024-06-15T00:55:13.595 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.517+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0116f30 con 0x7f66b00fb570 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.517+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66a00113a0 con 0x7f66b00fb570 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.517+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b0117440 con 0x7f66b00fb570 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.518+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b00fc920 con 0x7f66b00fb570 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.518+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f66a0012400 con 0x7f66b00fb570 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.519+0000 7f66acff9640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6698065430 0x7f66980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.519+0000 7f66af7fe640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6698065430 0x7f66980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.519+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f66a0082620 con 0x7f66b00fb570 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.519+0000 7f66af7fe640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6698065430 0x7f66980678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66a4004540 tx=0x7f66a4035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:13.596 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.523+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f66a0050c10 con 0x7f66b00fb570 2024-06-15T00:55:13.725 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.725+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f66b0072c20 con 0x7f66b00fb570 2024-06-15T00:55:13.726 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.725+0000 7f66acff9640 1 -- 172.21.15.105:0/873718426 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f66a0054230 con 0x7f66b00fb570 2024-06-15T00:55:13.726 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:13.726 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:13.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6698065430 msgr2=0x7f66980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:13.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6698065430 0x7f66980678e0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 crypto rx=0x7f66a4004540 tx=0x7f66a4035040 comp rx=0 tx=0).stop 2024-06-15T00:55:13.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 msgr2=0x7f66b010dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 0x7f66b010dec0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 crypto rx=0x7f66a000c790 tx=0x7f66a00043c0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 shutdown_connections 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6698065430 0x7f66980678e0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66b00fbe80 0x7f66b010e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 --2- 172.21.15.105:0/873718426 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66b00fb570 0x7f66b010dec0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 >> 172.21.15.105:0/873718426 conn(0x7f66b0072110 msgr2=0x7f66b00ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 shutdown_connections 2024-06-15T00:55:13.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:13.728+0000 7f66b6525640 1 -- 172.21.15.105:0/873718426 wait complete. 2024-06-15T00:55:14.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:13 smithi105 ceph-mon[28877]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:14.027 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:13 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/873718426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:14.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:13 smithi143 ceph-mon[35860]: pgmap v203: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:14.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:13 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/873718426' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:16.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:15 smithi105 ceph-mon[28877]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:16.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:15 smithi143 ceph-mon[35860]: pgmap v204: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:17.399 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:17.653 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:18.026 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:17 smithi105 ceph-mon[28877]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:18.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:17 smithi143 ceph-mon[35860]: pgmap v205: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 -- 172.21.15.105:0/2740641907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 msgr2=0x7f44400fae30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 --2- 172.21.15.105:0/2740641907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f44400fae30 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 crypto rx=0x7f44340048d0 tx=0x7f4434009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 -- 172.21.15.105:0/2740641907 shutdown_connections 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 --2- 172.21.15.105:0/2740641907 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f44400fae30 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 --2- 172.21.15.105:0/2740641907 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 0x7f4440102740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 -- 172.21.15.105:0/2740641907 >> 172.21.15.105:0/2740641907 conn(0x7f44400709e0 msgr2=0x7f4440070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 -- 172.21.15.105:0/2740641907 shutdown_connections 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.333+0000 7f4445538640 1 -- 172.21.15.105:0/2740641907 wait complete. 2024-06-15T00:55:19.341 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f4445538640 1 Processor -- start 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f4445538640 1 -- start start 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f4445538640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f4440112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f4445538640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 0x7f4440113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f4445538640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4440113b90 con 0x7f44400fa9c0 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f4445538640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f444010cf40 con 0x7f4440102370 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f443effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f4440112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f443effd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f4440112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59768/0 (socket says 172.21.15.105:59768) 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.334+0000 7f443effd640 1 -- 172.21.15.105:0/1344997692 learned_addr learned my addr 172.21.15.105:0/1344997692 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f443e7fc640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 0x7f4440113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:19.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f443effd640 1 -- 172.21.15.105:0/1344997692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 msgr2=0x7f4440113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f443effd640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 0x7f4440113500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f443effd640 1 -- 172.21.15.105:0/1344997692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4434005040 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f443e7fc640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 0x7f4440113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f443effd640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f4440112fc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f443000b790 tx=0x7f44300091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f443000c700 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f443000cd10 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f444010d220 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4430013370 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.335+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f444010d740 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.336+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4440065570 con 0x7f44400fa9c0 2024-06-15T00:55:19.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.336+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4430012070 con 0x7f44400fa9c0 2024-06-15T00:55:19.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.337+0000 7f44277fe640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4428065430 0x7f44280678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:19.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.337+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4430082170 con 0x7f44400fa9c0 2024-06-15T00:55:19.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.337+0000 7f443e7fc640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4428065430 0x7f44280678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:19.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.337+0000 7f443e7fc640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4428065430 0x7f44280678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4434007ac0 tx=0x7f4434035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:19.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.341+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4430050620 con 0x7f44400fa9c0 2024-06-15T00:55:19.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.539+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f444004b4b0 con 0x7f44400fa9c0 2024-06-15T00:55:19.540 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.540+0000 7f44277fe640 1 -- 172.21.15.105:0/1344997692 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4430053c40 con 0x7f44400fa9c0 2024-06-15T00:55:19.540 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:19.540 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:19.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.541+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4428065430 msgr2=0x7f44280678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:19.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.541+0000 7f4445538640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4428065430 0x7f44280678e0 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 crypto rx=0x7f4434007ac0 tx=0x7f4434035040 comp rx=0 tx=0).stop 2024-06-15T00:55:19.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.541+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 msgr2=0x7f4440112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:19.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.541+0000 7f4445538640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f4440112fc0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 crypto rx=0x7f443000b790 tx=0x7f44300091d0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 shutdown_connections 2024-06-15T00:55:19.542 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4428065430 0x7f44280678e0 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4440102370 0x7f4440113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 --2- 172.21.15.105:0/1344997692 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f44400fa9c0 0x7f4440112fc0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:19.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 >> 172.21.15.105:0/1344997692 conn(0x7f44400709e0 msgr2=0x7f44400f9db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:19.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 shutdown_connections 2024-06-15T00:55:19.543 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:19.542+0000 7f4445538640 1 -- 172.21.15.105:0/1344997692 wait complete. 2024-06-15T00:55:20.245 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:19 smithi105 ceph-mon[28877]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:20.245 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:19 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1344997692' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:20.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:19 smithi143 ceph-mon[35860]: pgmap v206: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:19 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1344997692' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:22.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:21 smithi105 ceph-mon[28877]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:22.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:21 smithi143 ceph-mon[35860]: pgmap v207: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:23.271 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:23.538 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:23 smithi105 ceph-mon[28877]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:24.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:23 smithi143 ceph-mon[35860]: pgmap v208: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:25.073 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:24 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:25.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 -- 172.21.15.105:0/2727355282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 msgr2=0x7f27981015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:25.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 --2- 172.21.15.105:0/2727355282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f27981015a0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 crypto rx=0x7f27900048d0 tx=0x7f2790009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 -- 172.21.15.105:0/2727355282 shutdown_connections 2024-06-15T00:55:25.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 --2- 172.21.15.105:0/2727355282 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2798101b70 0x7f27980fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.268 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 --2- 172.21.15.105:0/2727355282 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f27981015a0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 -- 172.21.15.105:0/2727355282 >> 172.21.15.105:0/2727355282 conn(0x7f27980f6dc0 msgr2=0x7f27980f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 -- 172.21.15.105:0/2727355282 shutdown_connections 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.266+0000 7f27a0292640 1 -- 172.21.15.105:0/2727355282 wait complete. 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f27a0292640 1 Processor -- start 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f27a0292640 1 -- start start 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f27a0292640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f279810ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f27a0292640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2798101b70 0x7f279810e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f27a0292640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279810ea60 con 0x7f27981011d0 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f27a0292640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f279810ebd0 con 0x7f2798101b70 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279e02e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f279810ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279e02e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f279810ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40446/0 (socket says 172.21.15.105:40446) 2024-06-15T00:55:25.269 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279e02e640 1 -- 172.21.15.105:0/1231612844 learned_addr learned my addr 172.21.15.105:0/1231612844 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279d82d640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2798101b70 0x7f279810e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279e02e640 1 -- 172.21.15.105:0/1231612844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2798101b70 msgr2=0x7f279810e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279e02e640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2798101b70 0x7f279810e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.267+0000 7f279e02e640 1 -- 172.21.15.105:0/1231612844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2790005040 con 0x7f27981011d0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.268+0000 7f279e02e640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f279810ded0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2790000c00 tx=0x7f2790009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.268+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2790038070 con 0x7f27981011d0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.268+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2798116f40 con 0x7f27981011d0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.268+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2790008ed0 con 0x7f27981011d0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.268+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2790032b40 con 0x7f27981011d0 2024-06-15T00:55:25.270 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.268+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27981173a0 con 0x7f27981011d0 2024-06-15T00:55:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.269+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27980fc940 con 0x7f27981011d0 2024-06-15T00:55:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.269+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2790030020 con 0x7f27981011d0 2024-06-15T00:55:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.270+0000 7f2796ffd640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2780065430 0x7f27800678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.270+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f27900a22f0 con 0x7f27981011d0 2024-06-15T00:55:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.270+0000 7f279d82d640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2780065430 0x7f27800678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:25.271 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.270+0000 7f279d82d640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2780065430 0x7f27800678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27980661d0 tx=0x7f278c004bb0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:25.274 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.274+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f27900707a0 con 0x7f27981011d0 2024-06-15T00:55:25.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:24 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:25.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.470+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f279804b480 con 0x7f27981011d0 2024-06-15T00:55:25.470 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.470+0000 7f2796ffd640 1 -- 172.21.15.105:0/1231612844 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f2790073dc0 con 0x7f27981011d0 2024-06-15T00:55:25.471 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:25.471 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:25.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2780065430 msgr2=0x7f27800678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:25.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2780065430 0x7f27800678e0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 crypto rx=0x7f27980661d0 tx=0x7f278c004bb0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 msgr2=0x7f279810ded0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:25.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f279810ded0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 crypto rx=0x7f2790000c00 tx=0x7f2790009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.473 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 shutdown_connections 2024-06-15T00:55:25.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2780065430 0x7f27800678e0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2798101b70 0x7f279810e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 --2- 172.21.15.105:0/1231612844 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f27981011d0 0x7f279810ded0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:25.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 >> 172.21.15.105:0/1231612844 conn(0x7f27980f6dc0 msgr2=0x7f27980f7e10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:25.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 shutdown_connections 2024-06-15T00:55:25.474 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:25.473+0000 7f27a0292640 1 -- 172.21.15.105:0/1231612844 wait complete. 2024-06-15T00:55:26.202 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:25 smithi105 ceph-mon[28877]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:26.202 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:25 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1231612844' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:26.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:25 smithi143 ceph-mon[35860]: pgmap v209: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:26.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:25 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1231612844' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:28.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:27 smithi105 ceph-mon[28877]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:28.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:27 smithi143 ceph-mon[35860]: pgmap v210: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:29.229 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:29.501 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:30.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:29 smithi105 ceph-mon[28877]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:30.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:29 smithi143 ceph-mon[35860]: pgmap v211: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:31.116 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 -- 172.21.15.105:0/4281601311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 msgr2=0x7f2380102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 --2- 172.21.15.105:0/4281601311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380102670 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 crypto rx=0x7f23780048d0 tx=0x7f2378009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 -- 172.21.15.105:0/4281601311 shutdown_connections 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 --2- 172.21.15.105:0/4281601311 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2380102c40 0x7f23800fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 --2- 172.21.15.105:0/4281601311 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380102670 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 -- 172.21.15.105:0/4281601311 >> 172.21.15.105:0/4281601311 conn(0x7f23800f7e70 msgr2=0x7f23800fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 -- 172.21.15.105:0/4281601311 shutdown_connections 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.114+0000 7f2388485640 1 -- 172.21.15.105:0/4281601311 wait complete. 2024-06-15T00:55:31.117 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2388485640 1 Processor -- start 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2388485640 1 -- start start 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2388485640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380115120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2388485640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2380102c40 0x7f2380115660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2388485640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2380115c60 con 0x7f23801022a0 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2388485640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f238010f0a0 con 0x7f2380102c40 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2386221640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380115120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2386221640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380115120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40460/0 (socket says 172.21.15.105:40460) 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2386221640 1 -- 172.21.15.105:0/3955698598 learned_addr learned my addr 172.21.15.105:0/3955698598 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.115+0000 7f2385a20640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2380102c40 0x7f2380115660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f2386221640 1 -- 172.21.15.105:0/3955698598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2380102c40 msgr2=0x7f2380115660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f2386221640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2380102c40 0x7f2380115660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.118 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f2386221640 1 -- 172.21.15.105:0/3955698598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2378005040 con 0x7f23801022a0 2024-06-15T00:55:31.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f2386221640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380115120 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f23780048a0 tx=0x7f2378009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:31.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2378038070 con 0x7f23801022a0 2024-06-15T00:55:31.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f237802cef0 con 0x7f23801022a0 2024-06-15T00:55:31.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f238010f320 con 0x7f23801022a0 2024-06-15T00:55:31.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f2378009640 con 0x7f23801022a0 2024-06-15T00:55:31.119 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.116+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f238010f7e0 con 0x7f23801022a0 2024-06-15T00:55:31.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.117+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23800fda10 con 0x7f23801022a0 2024-06-15T00:55:31.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.117+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f2378030020 con 0x7f23801022a0 2024-06-15T00:55:31.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.118+0000 7f237effd640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2368065430 0x7f23680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:31.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.118+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f23780a28e0 con 0x7f23801022a0 2024-06-15T00:55:31.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.118+0000 7f2385a20640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2368065430 0x7f23680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:31.120 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.118+0000 7f2385a20640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2368065430 0x7f23680678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23801109e0 tx=0x7f237400b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:31.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.122+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f2378070d90 con 0x7f23801022a0 2024-06-15T00:55:31.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.320+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f238004b480 con 0x7f23801022a0 2024-06-15T00:55:31.321 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.320+0000 7f237effd640 1 -- 172.21.15.105:0/3955698598 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f23780743b0 con 0x7f23801022a0 2024-06-15T00:55:31.321 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:31.321 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2368065430 msgr2=0x7f23680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2368065430 0x7f23680678e0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 crypto rx=0x7f23801109e0 tx=0x7f237400b040 comp rx=0 tx=0).stop 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 msgr2=0x7f2380115120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380115120 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 crypto rx=0x7f23780048a0 tx=0x7f2378009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 shutdown_connections 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f2368065430 0x7f23680678e0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f2380102c40 0x7f2380115660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 --2- 172.21.15.105:0/3955698598 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f23801022a0 0x7f2380115120 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 >> 172.21.15.105:0/3955698598 conn(0x7f23800f7e70 msgr2=0x7f23800f8db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:31.323 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 shutdown_connections 2024-06-15T00:55:31.324 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:31.322+0000 7f2388485640 1 -- 172.21.15.105:0/3955698598 wait complete. 2024-06-15T00:55:32.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:31 smithi105 ceph-mon[28877]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:32.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:31 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3955698598' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:32.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:31 smithi143 ceph-mon[35860]: pgmap v212: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:32.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:31 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3955698598' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:34.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:33 smithi105 ceph-mon[28877]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:33 smithi143 ceph-mon[35860]: pgmap v213: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:35.002 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:35.266 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:36.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:35 smithi105 ceph-mon[28877]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:36.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:35 smithi143 ceph-mon[35860]: pgmap v214: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:36.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 -- 172.21.15.105:0/3855161077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 msgr2=0x7f177c0701c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:36.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 --2- 172.21.15.105:0/3855161077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c0701c0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 crypto rx=0x7f176c0048d0 tx=0x7f176c009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:36.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 -- 172.21.15.105:0/3855161077 shutdown_connections 2024-06-15T00:55:36.888 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 --2- 172.21.15.105:0/3855161077 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c0701c0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 --2- 172.21.15.105:0/3855161077 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f177c071440 0x7f177c06f810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 -- 172.21.15.105:0/3855161077 >> 172.21.15.105:0/3855161077 conn(0x7f177c0f7e50 msgr2=0x7f177c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 -- 172.21.15.105:0/3855161077 shutdown_connections 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 -- 172.21.15.105:0/3855161077 wait complete. 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.887+0000 7f17840ed640 1 Processor -- start 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f17840ed640 1 -- start start 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f17840ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c1198c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f17840ed640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f177c071440 0x7f177c119e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f17840ed640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c11a4e0 con 0x7f177c06fd50 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f17840ed640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f177c113860 con 0x7f177c071440 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c1198c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c1198c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58992/0 (socket says 172.21.15.105:58992) 2024-06-15T00:55:36.889 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 -- 172.21.15.105:0/786529775 learned_addr learned my addr 172.21.15.105:0/786529775 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:36.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 -- 172.21.15.105:0/786529775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f177c071440 msgr2=0x7f177c119e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:36.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f177c071440 0x7f177c119e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:36.890 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 -- 172.21.15.105:0/786529775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c005040 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f1781e89640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c1198c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f177800b790 tx=0x7f17780091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.888+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f177800c700 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.889+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f177800cd10 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.889+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1778013370 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.889+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f177c113af0 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.889+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f177c114010 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.890+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f177c070f50 con 0x7f177c06fd50 2024-06-15T00:55:36.891 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.890+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1778012070 con 0x7f177c06fd50 2024-06-15T00:55:36.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.890+0000 7f17727fc640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1768065430 0x7f17680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:36.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.891+0000 7f1781688640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1768065430 0x7f17680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:36.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.891+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1778082030 con 0x7f177c06fd50 2024-06-15T00:55:36.892 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.891+0000 7f1781688640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1768065430 0x7f17680678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f176c032410 tx=0x7f176c009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:36.895 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:36.895+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1778050560 con 0x7f177c06fd50 2024-06-15T00:55:37.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.100+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f177c04b480 con 0x7f177c06fd50 2024-06-15T00:55:37.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.100+0000 7f17727fc640 1 -- 172.21.15.105:0/786529775 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1778053b80 con 0x7f177c06fd50 2024-06-15T00:55:37.100 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:37.100 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:37.103 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1768065430 msgr2=0x7f17680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1768065430 0x7f17680678e0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 crypto rx=0x7f176c032410 tx=0x7f176c009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 msgr2=0x7f177c1198c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c1198c0 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 crypto rx=0x7f177800b790 tx=0x7f17780091d0 comp rx=0 tx=0).stop 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 shutdown_connections 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1768065430 0x7f17680678e0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f177c071440 0x7f177c119e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 --2- 172.21.15.105:0/786529775 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f177c06fd50 0x7f177c1198c0 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:37.104 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 >> 172.21.15.105:0/786529775 conn(0x7f177c0f7e50 msgr2=0x7f177c0f90b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:37.105 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 shutdown_connections 2024-06-15T00:55:37.105 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:37.103+0000 7f17840ed640 1 -- 172.21.15.105:0/786529775 wait complete. 2024-06-15T00:55:38.231 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:37 smithi105 ceph-mon[28877]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:38.232 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:37 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/786529775' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:38.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:37 smithi143 ceph-mon[35860]: pgmap v215: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:38.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:37 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/786529775' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:39 smithi105 ceph-mon[28877]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:39 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:40.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:39 smithi143 ceph-mon[35860]: pgmap v216: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:40.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:39 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:40.816 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:41.085 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:42.244 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:41 smithi105 ceph-mon[28877]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:42.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:41 smithi143 ceph-mon[35860]: pgmap v217: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 -- 172.21.15.105:0/814999550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 msgr2=0x7f147c1026a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 --2- 172.21.15.105:0/814999550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c1026a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 crypto rx=0x7f146c0048d0 tx=0x7f146c009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 -- 172.21.15.105:0/814999550 shutdown_connections 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 --2- 172.21.15.105:0/814999550 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f147c102c70 0x7f147c0fc300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 --2- 172.21.15.105:0/814999550 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c1026a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 -- 172.21.15.105:0/814999550 >> 172.21.15.105:0/814999550 conn(0x7f147c0f7e80 msgr2=0x7f147c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 -- 172.21.15.105:0/814999550 shutdown_connections 2024-06-15T00:55:42.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.601+0000 7f1482080640 1 -- 172.21.15.105:0/814999550 wait complete. 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f1482080640 1 Processor -- start 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f1482080640 1 -- start start 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f1482080640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c112f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f1482080640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f147c102c70 0x7f147c113480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f1482080640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c113a80 con 0x7f147c1022d0 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f1482080640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f147c10cec0 con 0x7f147c102c70 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f147b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c112f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f147b7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c112f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:59008/0 (socket says 172.21.15.105:59008) 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f147b7fe640 1 -- 172.21.15.105:0/4269374841 learned_addr learned my addr 172.21.15.105:0/4269374841 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f147b7fe640 1 -- 172.21.15.105:0/4269374841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f147c102c70 msgr2=0x7f147c113480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f147b7fe640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f147c102c70 0x7f147c113480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.604 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.602+0000 7f147b7fe640 1 -- 172.21.15.105:0/4269374841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f146c005040 con 0x7f147c1022d0 2024-06-15T00:55:42.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.603+0000 7f147b7fe640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c112f40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f146c0048a0 tx=0x7f146c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:42.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.603+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c038070 con 0x7f147c1022d0 2024-06-15T00:55:42.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.603+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f146c032d30 con 0x7f147c1022d0 2024-06-15T00:55:42.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.603+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f147c10d140 con 0x7f147c1022d0 2024-06-15T00:55:42.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.603+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f146c009680 con 0x7f147c1022d0 2024-06-15T00:55:42.605 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.603+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f147c10d680 con 0x7f147c1022d0 2024-06-15T00:55:42.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.604+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f147c0fda40 con 0x7f147c1022d0 2024-06-15T00:55:42.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.604+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f146c030020 con 0x7f147c1022d0 2024-06-15T00:55:42.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.605+0000 7f1478ff9640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1468065430 0x7f14680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:42.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.605+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f146c0a2300 con 0x7f147c1022d0 2024-06-15T00:55:42.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.605+0000 7f1473fff640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1468065430 0x7f14680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:42.606 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.605+0000 7f1473fff640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1468065430 0x7f14680678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f147c10e800 tx=0x7f1464007cb0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:42.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.609+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f146c0707b0 con 0x7f147c1022d0 2024-06-15T00:55:42.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.809+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f147c10e5b0 con 0x7f147c1022d0 2024-06-15T00:55:42.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.809+0000 7f1478ff9640 1 -- 172.21.15.105:0/4269374841 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f146c073dd0 con 0x7f147c1022d0 2024-06-15T00:55:42.809 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:42.810 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:42.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1468065430 msgr2=0x7f14680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1468065430 0x7f14680678e0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 crypto rx=0x7f147c10e800 tx=0x7f1464007cb0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 msgr2=0x7f147c112f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c112f40 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 crypto rx=0x7f146c0048a0 tx=0x7f146c009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 shutdown_connections 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1468065430 0x7f14680678e0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f147c102c70 0x7f147c113480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 --2- 172.21.15.105:0/4269374841 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f147c1022d0 0x7f147c112f40 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 >> 172.21.15.105:0/4269374841 conn(0x7f147c0f7e80 msgr2=0x7f147c0f8ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 shutdown_connections 2024-06-15T00:55:42.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:42.811+0000 7f1482080640 1 -- 172.21.15.105:0/4269374841 wait complete. 2024-06-15T00:55:43.244 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:42 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/4269374841' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:43.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:42 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/4269374841' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:44.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:43 smithi105 ceph-mon[28877]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:44.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:43 smithi143 ceph-mon[35860]: pgmap v218: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:45 smithi105 ceph-mon[28877]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:46.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:45 smithi143 ceph-mon[35860]: pgmap v219: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:46.513 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:46.777 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:48.162 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:47 smithi105 ceph-mon[28877]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.341+0000 7fdeb49f7640 1 -- 172.21.15.105:0/10354052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 msgr2=0x7fdeb01015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.341+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/10354052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb01015a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 crypto rx=0x7fdea0009c50 tx=0x7fdea0004d10 comp rx=0 tx=0).stop 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.341+0000 7fdeb49f7640 1 -- 172.21.15.105:0/10354052 shutdown_connections 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/10354052 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdeb0101b70 0x7fdeb00fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/10354052 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb01015a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 -- 172.21.15.105:0/10354052 >> 172.21.15.105:0/10354052 conn(0x7fdeb00f6dc0 msgr2=0x7fdeb00f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 -- 172.21.15.105:0/10354052 shutdown_connections 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 -- 172.21.15.105:0/10354052 wait complete. 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 Processor -- start 2024-06-15T00:55:48.365 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 -- start start 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb0112f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdeb0101b70 0x7fdeb01134b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb0113b40 con 0x7fdeb01011d0 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeb49f7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb010cef0 con 0x7fdeb0101b70 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeae59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb0112f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeae59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb0112f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:48600/0 (socket says 172.21.15.105:48600) 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.342+0000 7fdeae59c640 1 -- 172.21.15.105:0/2665952576 learned_addr learned my addr 172.21.15.105:0/2665952576 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fdeae59c640 1 -- 172.21.15.105:0/2665952576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdeb0101b70 msgr2=0x7fdeb01134b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fdeae59c640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdeb0101b70 0x7fdeb01134b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fdeae59c640 1 -- 172.21.15.105:0/2665952576 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdea00098b0 con 0x7fdeb01011d0 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fdeae59c640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb0112f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdea0003400 tx=0x7fdea0004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdea0041070 con 0x7fdeb01011d0 2024-06-15T00:55:48.366 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb010d170 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdea0009690 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdea003d8c0 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.343+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb010d630 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.344+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb00fc940 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.344+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdea002f030 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.345+0000 7fde9effd640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fde98065430 0x7fde980678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.345+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdea00a3170 con 0x7fdeb01011d0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.345+0000 7fdeadd9b640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fde98065430 0x7fde980678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.345+0000 7fdeadd9b640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fde98065430 0x7fde980678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdeb010e830 tx=0x7fdea4006d20 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:48.367 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.349+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdea0071620 con 0x7fdeb01011d0 2024-06-15T00:55:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:47 smithi143 ceph-mon[35860]: pgmap v220: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:48.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.547+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdeb01082e0 con 0x7fdeb01011d0 2024-06-15T00:55:48.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.547+0000 7fde9effd640 1 -- 172.21.15.105:0/2665952576 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdea0074c40 con 0x7fdeb01011d0 2024-06-15T00:55:48.547 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:48.548 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fde98065430 msgr2=0x7fde980678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fde98065430 0x7fde980678e0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 crypto rx=0x7fdeb010e830 tx=0x7fdea4006d20 comp rx=0 tx=0).stop 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 msgr2=0x7fdeb0112f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb0112f70 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 crypto rx=0x7fdea0003400 tx=0x7fdea0004d10 comp rx=0 tx=0).stop 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 shutdown_connections 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fde98065430 0x7fde980678e0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:48.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdeb0101b70 0x7fdeb01134b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:48.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 --2- 172.21.15.105:0/2665952576 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdeb01011d0 0x7fdeb0112f70 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:48.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 >> 172.21.15.105:0/2665952576 conn(0x7fdeb00f6dc0 msgr2=0x7fdeb00f8520 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:48.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 shutdown_connections 2024-06-15T00:55:48.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:48.549+0000 7fdeb49f7640 1 -- 172.21.15.105:0/2665952576 wait complete. 2024-06-15T00:55:49.196 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:48 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2665952576' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:49.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:48 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2665952576' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:50.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:49 smithi105 ceph-mon[28877]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:50.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:49 smithi143 ceph-mon[35860]: pgmap v221: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:52.223 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:52.247 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:51 smithi105 ceph-mon[28877]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:52.247 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:55:52.247 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:55:52.247 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:55:52.247 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:51 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:55:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:51 smithi143 ceph-mon[35860]: pgmap v222: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:55:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:55:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:55:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:51 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:55:52.476 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:54.042 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 -- 172.21.15.105:0/4140887659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c10102370 msgr2=0x7f7c10102740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 --2- 172.21.15.105:0/4140887659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c10102370 0x7f7c10102740 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 crypto rx=0x7f7c000048d0 tx=0x7f7c00009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 -- 172.21.15.105:0/4140887659 shutdown_connections 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 --2- 172.21.15.105:0/4140887659 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c100fa9c0 0x7f7c100fae30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 --2- 172.21.15.105:0/4140887659 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c10102370 0x7f7c10102740 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 -- 172.21.15.105:0/4140887659 >> 172.21.15.105:0/4140887659 conn(0x7f7c100709e0 msgr2=0x7f7c10070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 -- 172.21.15.105:0/4140887659 shutdown_connections 2024-06-15T00:55:54.043 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.042+0000 7f7c18046640 1 -- 172.21.15.105:0/4140887659 wait complete. 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c18046640 1 Processor -- start 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c18046640 1 -- start start 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c18046640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 0x7f7c10112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c18046640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10102370 0x7f7c101134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c18046640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c10113b70 con 0x7f7c100fa9c0 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c18046640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c1010cf20 con 0x7f7c10102370 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c15de2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 0x7f7c10112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c15de2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 0x7f7c10112fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55854/0 (socket says 172.21.15.105:55854) 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.043+0000 7f7c15de2640 1 -- 172.21.15.105:0/3045243597 learned_addr learned my addr 172.21.15.105:0/3045243597 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c155e1640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10102370 0x7f7c101134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:54.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c15de2640 1 -- 172.21.15.105:0/3045243597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10102370 msgr2=0x7f7c101134e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c15de2640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10102370 0x7f7c101134e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c15de2640 1 -- 172.21.15.105:0/3045243597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c00005040 con 0x7f7c100fa9c0 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c155e1640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10102370 0x7f7c101134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c15de2640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 0x7f7c10112fa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7c000048a0 tx=0x7f7c00009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c00038070 con 0x7f7c100fa9c0 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c1010d1a0 con 0x7f7c100fa9c0 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c00032d30 con 0x7f7c100fa9c0 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7c00009680 con 0x7f7c100fa9c0 2024-06-15T00:55:54.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.044+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c1010d660 con 0x7f7c100fa9c0 2024-06-15T00:55:54.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.045+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c10065570 con 0x7f7c100fa9c0 2024-06-15T00:55:54.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.045+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7c00030020 con 0x7f7c100fa9c0 2024-06-15T00:55:54.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.046+0000 7f7c067fc640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7bfc065430 0x7f7bfc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:54.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.046+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f7c000a2300 con 0x7f7c100fa9c0 2024-06-15T00:55:54.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.046+0000 7f7c155e1640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7bfc065430 0x7f7bfc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:54.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.046+0000 7f7c155e1640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7bfc065430 0x7f7bfc0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c006fd0 tx=0x7f7c0c008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:54.050 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.050+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7c000707b0 con 0x7f7c100fa9c0 2024-06-15T00:55:54.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.249+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f7c1004b510 con 0x7f7c100fa9c0 2024-06-15T00:55:54.250 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:54 smithi105 ceph-mon[28877]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:54.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.250+0000 7f7c067fc640 1 -- 172.21.15.105:0/3045243597 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7c00073dd0 con 0x7f7c100fa9c0 2024-06-15T00:55:54.251 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:55:54.251 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:55:54.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7bfc065430 msgr2=0x7f7bfc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:54.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7bfc065430 0x7f7bfc0678e0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 crypto rx=0x7f7c0c006fd0 tx=0x7f7c0c008040 comp rx=0 tx=0).stop 2024-06-15T00:55:54.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 msgr2=0x7f7c10112fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:54.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 0x7f7c10112fa0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 crypto rx=0x7f7c000048a0 tx=0x7f7c00009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 shutdown_connections 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7bfc065430 0x7f7bfc0678e0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7c10102370 0x7f7c101134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 --2- 172.21.15.105:0/3045243597 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f7c100fa9c0 0x7f7c10112fa0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 >> 172.21.15.105:0/3045243597 conn(0x7f7c100709e0 msgr2=0x7f7c100f9d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 shutdown_connections 2024-06-15T00:55:54.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:54.252+0000 7f7c18046640 1 -- 172.21.15.105:0/3045243597 wait complete. 2024-06-15T00:55:54.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:54 smithi143 ceph-mon[35860]: pgmap v223: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:55.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:55 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3045243597' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:55.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:55.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:55 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3045243597' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:55:55.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:55:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:56 smithi105 ceph-mon[28877]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:56.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:56 smithi143 ceph-mon[35860]: pgmap v224: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:57.937 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:55:58.202 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:55:58.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:55:58 smithi105 ceph-mon[28877]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:58.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:55:58 smithi143 ceph-mon[35860]: pgmap v225: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:55:59.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 -- 172.21.15.105:0/2823854662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94241011b0 msgr2=0x7f9424101580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:59.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 --2- 172.21.15.105:0/2823854662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94241011b0 0x7f9424101580 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 crypto rx=0x7f94200048d0 tx=0x7f9420009da0 comp rx=0 tx=0).stop 2024-06-15T00:55:59.847 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 -- 172.21.15.105:0/2823854662 shutdown_connections 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 --2- 172.21.15.105:0/2823854662 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f9424101b50 0x7f94240fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 --2- 172.21.15.105:0/2823854662 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f94241011b0 0x7f9424101580 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 -- 172.21.15.105:0/2823854662 >> 172.21.15.105:0/2823854662 conn(0x7f94240f6d60 msgr2=0x7f94240f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 -- 172.21.15.105:0/2823854662 shutdown_connections 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.839+0000 7f942c617640 1 -- 172.21.15.105:0/2823854662 wait complete. 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942c617640 1 Processor -- start 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942c617640 1 -- start start 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942c617640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94241011b0 0x7f9424112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942c617640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 0x7f94241134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942c617640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9424113b70 con 0x7f9424101b50 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942c617640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f942410cf20 con 0x7f94241011b0 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f9429bb2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 0x7f94241134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:59.848 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f9429bb2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 0x7f94241134e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:55864/0 (socket says 172.21.15.105:55864) 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f9429bb2640 1 -- 172.21.15.105:0/432172523 learned_addr learned my addr 172.21.15.105:0/432172523 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f942a3b3640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94241011b0 0x7f9424112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f9429bb2640 1 -- 172.21.15.105:0/432172523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94241011b0 msgr2=0x7f9424112fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f9429bb2640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94241011b0 0x7f9424112fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.840+0000 7f9429bb2640 1 -- 172.21.15.105:0/432172523 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9420005040 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.841+0000 7f9429bb2640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 0x7f94241134e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f941400b790 tx=0x7f94140091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.841+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f941400c700 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.841+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f941400cd10 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.841+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f9414013370 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.841+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f942410d200 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.841+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f942410d6d0 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.842+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f94240fc920 con 0x7f9424101b50 2024-06-15T00:55:59.849 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.842+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f9414012070 con 0x7f9424101b50 2024-06-15T00:55:59.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.843+0000 7f941affd640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9410065430 0x7f94100678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:55:59.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.843+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f9414082170 con 0x7f9424101b50 2024-06-15T00:55:59.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.843+0000 7f942a3b3640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9410065430 0x7f94100678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:55:59.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.843+0000 7f942a3b3640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9410065430 0x7f94100678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9420004540 tx=0x7f9420035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:55:59.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:55:59.847+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f9414050620 con 0x7f9424101b50 2024-06-15T00:56:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.044+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f9424109360 con 0x7f9424101b50 2024-06-15T00:56:00.045 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.044+0000 7f941affd640 1 -- 172.21.15.105:0/432172523 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f9414053c40 con 0x7f9424101b50 2024-06-15T00:56:00.045 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:00.045 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:00.046 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9410065430 msgr2=0x7f94100678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9410065430 0x7f94100678e0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 crypto rx=0x7f9420004540 tx=0x7f9420035040 comp rx=0 tx=0).stop 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 msgr2=0x7f94241134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 0x7f94241134e0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 crypto rx=0x7f941400b790 tx=0x7f94140091d0 comp rx=0 tx=0).stop 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 shutdown_connections 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f9410065430 0x7f94100678e0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f9424101b50 0x7f94241134e0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 --2- 172.21.15.105:0/432172523 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f94241011b0 0x7f9424112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 >> 172.21.15.105:0/432172523 conn(0x7f94240f6d60 msgr2=0x7f94240f7d30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 shutdown_connections 2024-06-15T00:56:00.047 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:00.046+0000 7f942c617640 1 -- 172.21.15.105:0/432172523 wait complete. 2024-06-15T00:56:00.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:00 smithi105 ceph-mon[28877]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:00.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:00 smithi143 ceph-mon[35860]: pgmap v226: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:01.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:01 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/432172523' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:01.287 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:01 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/432172523' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:02.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:02 smithi105 ceph-mon[28877]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:02.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:02 smithi143 ceph-mon[35860]: pgmap v227: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:03.629 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:03.896 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:04.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:04 smithi105 ceph-mon[28877]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:04.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:04 smithi143 ceph-mon[35860]: pgmap v228: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:05.462 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- 172.21.15.105:0/339016647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc101b60 msgr2=0x7f32cc0fb1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:05.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 --2- 172.21.15.105:0/339016647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc101b60 0x7f32cc0fb1f0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 crypto rx=0x7f32b40048d0 tx=0x7f32b4009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.465 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- 172.21.15.105:0/339016647 shutdown_connections 2024-06-15T00:56:05.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 --2- 172.21.15.105:0/339016647 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc101b60 0x7f32cc0fb1f0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 --2- 172.21.15.105:0/339016647 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc1011c0 0x7f32cc101590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- 172.21.15.105:0/339016647 >> 172.21.15.105:0/339016647 conn(0x7f32cc0f6d70 msgr2=0x7f32cc0f9180 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:05.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- 172.21.15.105:0/339016647 shutdown_connections 2024-06-15T00:56:05.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- 172.21.15.105:0/339016647 wait complete. 2024-06-15T00:56:05.466 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 Processor -- start 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- start start 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 0x7f32cc112ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101b60 0x7f32cc113430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc113a30 con 0x7f32cc1011c0 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.454+0000 7f32d0e07640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32cc113ba0 con 0x7f32cc101b60 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 0x7f32cc112ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 0x7f32cc112ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47088/0 (socket says 172.21.15.105:47088) 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 -- 172.21.15.105:0/2992286737 learned_addr learned my addr 172.21.15.105:0/2992286737 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 -- 172.21.15.105:0/2992286737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101b60 msgr2=0x7f32cc113430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32c3fff640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101b60 0x7f32cc113430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:05.467 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101b60 0x7f32cc113430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 -- 172.21.15.105:0/2992286737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32b4005040 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32ca59c640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 0x7f32cc112ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00c790 tx=0x7f32bc0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32bc00a820 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32cc10d020 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32bc00ae30 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f32bc0113a0 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.455+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32cc10d5c0 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.456+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32cc0fc930 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.457+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f32bc012400 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.457+0000 7f32c2ffd640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f32b8065430 0x7f32b80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.457+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f32bc0826c0 con 0x7f32cc1011c0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.457+0000 7f32c3fff640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f32b8065430 0x7f32b80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:05.468 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.458+0000 7f32c3fff640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f32b8065430 0x7f32b80678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f32b4003080 tx=0x7f32b4009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:05.469 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.461+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f32bc050bf0 con 0x7f32cc1011c0 2024-06-15T00:56:05.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.658+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f32cc10e5a0 con 0x7f32cc1011c0 2024-06-15T00:56:05.659 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.659+0000 7f32c2ffd640 1 -- 172.21.15.105:0/2992286737 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f32bc00a980 con 0x7f32cc1011c0 2024-06-15T00:56:05.659 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:05.659 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:05.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f32b8065430 msgr2=0x7f32b80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:05.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f32b8065430 0x7f32b80678e0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 crypto rx=0x7f32b4003080 tx=0x7f32b4009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 msgr2=0x7f32cc112ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:05.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 0x7f32cc112ef0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 crypto rx=0x7f32bc00c790 tx=0x7f32bc0043c0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 shutdown_connections 2024-06-15T00:56:05.661 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f32b8065430 0x7f32b80678e0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f32cc101b60 0x7f32cc113430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 --2- 172.21.15.105:0/2992286737 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f32cc1011c0 0x7f32cc112ef0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:05.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.660+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 >> 172.21.15.105:0/2992286737 conn(0x7f32cc0f6d70 msgr2=0x7f32cc065870 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:05.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.661+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 shutdown_connections 2024-06-15T00:56:05.662 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:05.661+0000 7f32d0e07640 1 -- 172.21.15.105:0/2992286737 wait complete. 2024-06-15T00:56:06.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:06 smithi105 ceph-mon[28877]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:06.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:06 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2992286737' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:06.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:06 smithi143 ceph-mon[35860]: pgmap v229: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:06.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:06 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2992286737' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:08.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:08 smithi143 ceph-mon[35860]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:08.530 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:08 smithi105 ceph-mon[28877]: pgmap v230: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:09.554 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:09.820 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:10.261 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:10 smithi105 ceph-mon[28877]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:10.261 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:10.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:10 smithi143 ceph-mon[35860]: pgmap v231: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:10.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:11.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 -- 172.21.15.105:0/135762830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc102360 msgr2=0x7fdcfc102730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:11.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 --2- 172.21.15.105:0/135762830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc102360 0x7fdcfc102730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0048d0 tx=0x7fdcec009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.402 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 -- 172.21.15.105:0/135762830 shutdown_connections 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 --2- 172.21.15.105:0/135762830 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc0fad80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 --2- 172.21.15.105:0/135762830 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc102360 0x7fdcfc102730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 -- 172.21.15.105:0/135762830 >> 172.21.15.105:0/135762830 conn(0x7fdcfc071820 msgr2=0x7fdcfc071c20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 -- 172.21.15.105:0/135762830 shutdown_connections 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 -- 172.21.15.105:0/135762830 wait complete. 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 Processor -- start 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.395+0000 7fdd01cac640 1 -- start start 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdd01cac640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdd01cac640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc102360 0x7fdcfc113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:11.403 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdd01cac640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc113b90 con 0x7fdcfc0fa930 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdd01cac640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcfc10cf40 con 0x7fdcfc102360 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:47102/0 (socket says 172.21.15.105:47102) 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 -- 172.21.15.105:0/2943400341 learned_addr learned my addr 172.21.15.105:0/2943400341 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfaffd640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc102360 0x7fdcfc113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 -- 172.21.15.105:0/2943400341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc102360 msgr2=0x7fdcfc113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc102360 0x7fdcfc113500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 -- 172.21.15.105:0/2943400341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec005040 con 0x7fdcfc0fa930 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfaffd640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc102360 0x7fdcfc113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdcfb7fe640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc112fc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0048a0 tx=0x7fdcec009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:11.404 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcec038070 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdcec032d30 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcfc10d1c0 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdcec009680 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.396+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcfc10d680 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.398+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcfc0fdc50 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.398+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdcec030020 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.398+0000 7fdce3fff640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdce4065430 0x7fdce40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.398+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdcec0a2260 con 0x7fdcfc0fa930 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.398+0000 7fdcfaffd640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdce4065430 0x7fdce40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:11.405 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.399+0000 7fdcfaffd640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdce4065430 0x7fdce40678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc10e880 tx=0x7fdcf0008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:11.406 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.403+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdcec070710 con 0x7fdcfc0fa930 2024-06-15T00:56:11.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.611+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdcfc04b4e0 con 0x7fdcfc0fa930 2024-06-15T00:56:11.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.611+0000 7fdce3fff640 1 -- 172.21.15.105:0/2943400341 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdcec073d30 con 0x7fdcfc0fa930 2024-06-15T00:56:11.612 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:11.613 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:11.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdce4065430 msgr2=0x7fdce40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:11.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdce4065430 0x7fdce40678e0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 crypto rx=0x7fdcfc10e880 tx=0x7fdcf0008040 comp rx=0 tx=0).stop 2024-06-15T00:56:11.614 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 msgr2=0x7fdcfc112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc112fc0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 crypto rx=0x7fdcec0048a0 tx=0x7fdcec009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 shutdown_connections 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdce4065430 0x7fdce40678e0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdcfc102360 0x7fdcfc113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 --2- 172.21.15.105:0/2943400341 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdcfc0fa930 0x7fdcfc112fc0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 >> 172.21.15.105:0/2943400341 conn(0x7fdcfc071820 msgr2=0x7fdcfc1068c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 shutdown_connections 2024-06-15T00:56:11.615 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:11.614+0000 7fdd01cac640 1 -- 172.21.15.105:0/2943400341 wait complete. 2024-06-15T00:56:12.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:12 smithi105 ceph-mon[28877]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:12.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:12 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2943400341' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:12.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:12 smithi143 ceph-mon[35860]: pgmap v232: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:12.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:12 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2943400341' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:14.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:14 smithi143 ceph-mon[35860]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:14.483 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:14 smithi105 ceph-mon[28877]: pgmap v233: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:15.334 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:15.600 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:16.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:16 smithi105 ceph-mon[28877]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:16.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:16 smithi143 ceph-mon[35860]: pgmap v234: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:17.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 -- 172.21.15.105:0/1049339521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 msgr2=0x7f47f4102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:17.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 --2- 172.21.15.105:0/1049339521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f4102690 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 crypto rx=0x7f47e80048d0 tx=0x7f47e8009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.409 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 -- 172.21.15.105:0/1049339521 shutdown_connections 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 --2- 172.21.15.105:0/1049339521 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f4102c60 0x7f47f40fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 --2- 172.21.15.105:0/1049339521 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f4102690 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 -- 172.21.15.105:0/1049339521 >> 172.21.15.105:0/1049339521 conn(0x7f47f40f7e50 msgr2=0x7f47f40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 -- 172.21.15.105:0/1049339521 shutdown_connections 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.403+0000 7f47fac3d640 1 -- 172.21.15.105:0/1049339521 wait complete. 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47fac3d640 1 Processor -- start 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47fac3d640 1 -- start start 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47fac3d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f41153f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47fac3d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f4102c60 0x7f47f4115930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47fac3d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f4115f30 con 0x7f47f41022c0 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47fac3d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47f41160a0 con 0x7f47f4102c60 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47f89d9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f41153f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:17.410 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47f89d9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f41153f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:35564/0 (socket says 172.21.15.105:35564) 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.404+0000 7f47f89d9640 1 -- 172.21.15.105:0/3367146086 learned_addr learned my addr 172.21.15.105:0/3367146086 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f3fff640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f4102c60 0x7f47f4115930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f89d9640 1 -- 172.21.15.105:0/3367146086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f4102c60 msgr2=0x7f47f4115930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f89d9640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f4102c60 0x7f47f4115930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f89d9640 1 -- 172.21.15.105:0/3367146086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47e8005040 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f89d9640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f41153f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f47e80048a0 tx=0x7f47e8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47e8038070 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f47e802cef0 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47f410f4c0 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f47e8009640 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.405+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47f410fa00 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.406+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47f40fda30 con 0x7f47f41022c0 2024-06-15T00:56:17.411 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.406+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f47e8030020 con 0x7f47f41022c0 2024-06-15T00:56:17.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.407+0000 7f47f17fa640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f47e4065430 0x7f47e40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:17.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.407+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f47e80a2930 con 0x7f47f41022c0 2024-06-15T00:56:17.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.407+0000 7f47f3fff640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f47e4065430 0x7f47e40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:17.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.407+0000 7f47f3fff640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f47e4065430 0x7f47e40678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007fd0 tx=0x7f47e0009e90 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:17.412 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.411+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f47e8070de0 con 0x7f47f41022c0 2024-06-15T00:56:17.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.608+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f47f40fbab0 con 0x7f47f41022c0 2024-06-15T00:56:17.609 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.609+0000 7f47f17fa640 1 -- 172.21.15.105:0/3367146086 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f47e8074400 con 0x7f47f41022c0 2024-06-15T00:56:17.609 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:17.609 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:17.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f47e4065430 msgr2=0x7f47e40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:17.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f47e4065430 0x7f47e40678e0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 crypto rx=0x7f47e0007fd0 tx=0x7f47e0009e90 comp rx=0 tx=0).stop 2024-06-15T00:56:17.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 msgr2=0x7f47f41153f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:17.611 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f41153f0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 crypto rx=0x7f47e80048a0 tx=0x7f47e8009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 shutdown_connections 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f47e4065430 0x7f47e40678e0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47f4102c60 0x7f47f4115930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 --2- 172.21.15.105:0/3367146086 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47f41022c0 0x7f47f41153f0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 >> 172.21.15.105:0/3367146086 conn(0x7f47f40f7e50 msgr2=0x7f47f40f8dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 shutdown_connections 2024-06-15T00:56:17.612 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:17.611+0000 7f47fac3d640 1 -- 172.21.15.105:0/3367146086 wait complete. 2024-06-15T00:56:18.155 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:18 smithi105 ceph-mon[28877]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:18.155 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:18 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3367146086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:18.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:18 smithi143 ceph-mon[35860]: pgmap v235: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:18.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:18 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3367146086' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:20.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:20 smithi143 ceph-mon[35860]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:20.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:20 smithi105 ceph-mon[28877]: pgmap v236: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:21.179 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:21.445 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:22.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:22 smithi105 ceph-mon[28877]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:22.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:22 smithi143 ceph-mon[35860]: pgmap v237: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:23.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 -- 172.21.15.105:0/1412306526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 msgr2=0x7f13fc065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 --2- 172.21.15.105:0/1412306526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc065470 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 crypto rx=0x7f13f80048d0 tx=0x7f13f8009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 -- 172.21.15.105:0/1412306526 shutdown_connections 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 --2- 172.21.15.105:0/1412306526 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13fc0fdeb0 0x7f13fc0fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 --2- 172.21.15.105:0/1412306526 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc065470 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 -- 172.21.15.105:0/1412306526 >> 172.21.15.105:0/1412306526 conn(0x7f13fc072110 msgr2=0x7f13fc072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 -- 172.21.15.105:0/1412306526 shutdown_connections 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 -- 172.21.15.105:0/1412306526 wait complete. 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 Processor -- start 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.044+0000 7f1404322640 1 -- start start 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f1404322640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f1404322640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13fc0fdeb0 0x7f13fc113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:23.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f1404322640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc113b90 con 0x7f13fc0650a0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f1404322640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13fc10cf40 con 0x7f13fc0fdeb0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:46626/0 (socket says 172.21.15.105:46626) 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 -- 172.21.15.105:0/81397292 learned_addr learned my addr 172.21.15.105:0/81397292 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 -- 172.21.15.105:0/81397292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13fc0fdeb0 msgr2=0x7f13fc113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13fc0fdeb0 0x7f13fc113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 -- 172.21.15.105:0/81397292 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13f8005040 con 0x7f13fc0650a0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f14020be640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc112fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f13f8000c00 tx=0x7f13f8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f8038070 con 0x7f13fc0650a0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13fc10d1c0 con 0x7f13fc0650a0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13f80096f0 con 0x7f13fc0650a0 2024-06-15T00:56:23.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f13f8032da0 con 0x7f13fc0650a0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.045+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13fc10d680 con 0x7f13fc0650a0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.046+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13fc0fda50 con 0x7f13fc0650a0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.047+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f13f8030020 con 0x7f13fc0650a0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.047+0000 7f13f2ffd640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f13e8065430 0x7f13e80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.047+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f13f80a2ab0 con 0x7f13fc0650a0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.047+0000 7f14018bd640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f13e8065430 0x7f13e80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.048+0000 7f14018bd640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f13e8065430 0x7f13e80678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f13fc10e880 tx=0x7f13ec006d20 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:23.059 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.051+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f13f8070f60 con 0x7f13fc0650a0 2024-06-15T00:56:23.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.249+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f13fc072c20 con 0x7f13fc0650a0 2024-06-15T00:56:23.250 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.250+0000 7f13f2ffd640 1 -- 172.21.15.105:0/81397292 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f13f8074580 con 0x7f13fc0650a0 2024-06-15T00:56:23.251 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:23.251 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:23.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f13e8065430 msgr2=0x7f13e80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:23.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f13e8065430 0x7f13e80678e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 crypto rx=0x7f13fc10e880 tx=0x7f13ec006d20 comp rx=0 tx=0).stop 2024-06-15T00:56:23.252 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 msgr2=0x7f13fc112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc112fc0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 crypto rx=0x7f13f8000c00 tx=0x7f13f8009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 shutdown_connections 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f13e8065430 0x7f13e80678e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f13fc0fdeb0 0x7f13fc113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 --2- 172.21.15.105:0/81397292 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f13fc0650a0 0x7f13fc112fc0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 >> 172.21.15.105:0/81397292 conn(0x7f13fc072110 msgr2=0x7f13fc0f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 shutdown_connections 2024-06-15T00:56:23.253 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:23.252+0000 7f1404322640 1 -- 172.21.15.105:0/81397292 wait complete. 2024-06-15T00:56:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:24 smithi105 ceph-mon[28877]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:24.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:24 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/81397292' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:24.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:24 smithi143 ceph-mon[35860]: pgmap v238: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:24.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:24 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/81397292' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:25.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:25.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:26.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:26 smithi143 ceph-mon[35860]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:26.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:26 smithi105 ceph-mon[28877]: pgmap v239: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:27.006 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:27.273 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:28.354 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:28 smithi105 ceph-mon[28877]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:28.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:28 smithi143 ceph-mon[35860]: pgmap v240: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:28.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 -- 172.21.15.105:0/1686255714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c1022a0 msgr2=0x7f661c102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:28.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 --2- 172.21.15.105:0/1686255714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c1022a0 0x7f661c102670 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 crypto rx=0x7f660c0048d0 tx=0x7f660c009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:28.826 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 -- 172.21.15.105:0/1686255714 shutdown_connections 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 --2- 172.21.15.105:0/1686255714 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c102c40 0x7f661c0fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 --2- 172.21.15.105:0/1686255714 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c1022a0 0x7f661c102670 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 -- 172.21.15.105:0/1686255714 >> 172.21.15.105:0/1686255714 conn(0x7f661c0f7e70 msgr2=0x7f661c0fa280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 -- 172.21.15.105:0/1686255714 shutdown_connections 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.824+0000 7f66209b7640 1 -- 172.21.15.105:0/1686255714 wait complete. 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f66209b7640 1 Processor -- start 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f66209b7640 1 -- start start 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f66209b7640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c1022a0 0x7f661c112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f66209b7640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 0x7f661c113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f66209b7640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c113b90 con 0x7f661c102c40 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f66209b7640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f661c10cf80 con 0x7f661c1022a0 2024-06-15T00:56:28.827 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f6611d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 0x7f661c113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f6611d9b640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 0x7f661c113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:46642/0 (socket says 172.21.15.105:46642) 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f6611d9b640 1 -- 172.21.15.105:0/1077416807 learned_addr learned my addr 172.21.15.105:0/1077416807 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.825+0000 7f661a59c640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c1022a0 0x7f661c112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f6611d9b640 1 -- 172.21.15.105:0/1077416807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c1022a0 msgr2=0x7f661c112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f6611d9b640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c1022a0 0x7f661c112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f6611d9b640 1 -- 172.21.15.105:0/1077416807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f660c005040 con 0x7f661c102c40 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f6611d9b640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 0x7f661c113500 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f660400b790 tx=0x7f66040091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f660400c700 con 0x7f661c102c40 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f660400cd10 con 0x7f661c102c40 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f6604013370 con 0x7f661c102c40 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f661c10d260 con 0x7f661c102c40 2024-06-15T00:56:28.828 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.826+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f661c10d780 con 0x7f661c102c40 2024-06-15T00:56:28.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.827+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f661c0fda10 con 0x7f661c102c40 2024-06-15T00:56:28.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.827+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f6604012070 con 0x7f661c102c40 2024-06-15T00:56:28.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.828+0000 7f66137fe640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6608065430 0x7f66080678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:28.829 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.828+0000 7f661a59c640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6608065430 0x7f66080678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:28.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.828+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f6604081ec0 con 0x7f661c102c40 2024-06-15T00:56:28.830 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.828+0000 7f661a59c640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6608065430 0x7f66080678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f660c004540 tx=0x7f660c035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:28.832 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:28.832+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f6604050370 con 0x7f661c102c40 2024-06-15T00:56:29.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.028+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f661c0fba90 con 0x7f661c102c40 2024-06-15T00:56:29.029 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.028+0000 7f66137fe640 1 -- 172.21.15.105:0/1077416807 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f6604053990 con 0x7f661c102c40 2024-06-15T00:56:29.029 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:29.029 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6608065430 msgr2=0x7f66080678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6608065430 0x7f66080678e0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 crypto rx=0x7f660c004540 tx=0x7f660c035040 comp rx=0 tx=0).stop 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 msgr2=0x7f661c113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 0x7f661c113500 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 crypto rx=0x7f660400b790 tx=0x7f66040091d0 comp rx=0 tx=0).stop 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 shutdown_connections 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6608065430 0x7f66080678e0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f661c102c40 0x7f661c113500 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 --2- 172.21.15.105:0/1077416807 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f661c1022a0 0x7f661c112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 >> 172.21.15.105:0/1077416807 conn(0x7f661c0f7e70 msgr2=0x7f661c0f8e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 shutdown_connections 2024-06-15T00:56:29.031 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:29.030+0000 7f66209b7640 1 -- 172.21.15.105:0/1077416807 wait complete. 2024-06-15T00:56:29.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:29 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1077416807' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:29.417 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:29 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1077416807' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:30.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:30 smithi143 ceph-mon[35860]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:30.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:30 smithi105 ceph-mon[28877]: pgmap v241: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:32.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:32 smithi143 ceph-mon[35860]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:32.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:32 smithi105 ceph-mon[28877]: pgmap v242: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:32.704 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:32.969 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:34.382 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:34 smithi105 ceph-mon[28877]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:34.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:34 smithi143 ceph-mon[35860]: pgmap v243: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 -- 172.21.15.105:0/999006298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c0faa50 msgr2=0x7f707c0faec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 --2- 172.21.15.105:0/999006298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c0faa50 0x7f707c0faec0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 crypto rx=0x7f70780048d0 tx=0x7f7078009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 -- 172.21.15.105:0/999006298 shutdown_connections 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 --2- 172.21.15.105:0/999006298 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c0faa50 0x7f707c0faec0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 --2- 172.21.15.105:0/999006298 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c1023e0 0x7f707c1027b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 -- 172.21.15.105:0/999006298 >> 172.21.15.105:0/999006298 conn(0x7f707c070ab0 msgr2=0x7f707c070eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 -- 172.21.15.105:0/999006298 shutdown_connections 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.615+0000 7f7083fb2640 1 -- 172.21.15.105:0/999006298 wait complete. 2024-06-15T00:56:34.616 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7083fb2640 1 Processor -- start 2024-06-15T00:56:34.617 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7083fb2640 1 -- start start 2024-06-15T00:56:34.618 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7083fb2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c0faa50 0x7f707c113080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7083fb2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 0x7f707c1135c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7083fb2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c113c50 con 0x7f707c1023e0 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7083fb2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f707c10d000 con 0x7f707c0faa50 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 0x7f707c1135c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 0x7f707c1135c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58722/0 (socket says 172.21.15.105:58722) 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 -- 172.21.15.105:0/1746865453 learned_addr learned my addr 172.21.15.105:0/1746865453 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7081d4e640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c0faa50 0x7f707c113080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 -- 172.21.15.105:0/1746865453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c0faa50 msgr2=0x7f707c113080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c0faa50 0x7f707c113080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 -- 172.21.15.105:0/1746865453 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7078005040 con 0x7f707c1023e0 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f7081d4e640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c0faa50 0x7f707c113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:56:34.619 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.616+0000 7f708154d640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 0x7f707c1135c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f70780048a0 tx=0x7f7078009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.617+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7078038070 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.617+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f707c10d280 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.617+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7078032d30 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.617+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7078009680 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.617+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f707c10d740 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.618+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f707c065550 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.618+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7078030020 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.618+0000 7f70727fc640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7068065430 0x7f70680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.619+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f70780a2260 con 0x7f707c1023e0 2024-06-15T00:56:34.620 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.619+0000 7f7081d4e640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7068065430 0x7f70680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:34.621 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.619+0000 7f7081d4e640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7068065430 0x7f70680678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f706c006fd0 tx=0x7f706c008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:34.623 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.623+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7078070710 con 0x7f707c1023e0 2024-06-15T00:56:34.822 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.822+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f707c0749d0 con 0x7f707c1023e0 2024-06-15T00:56:34.823 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.822+0000 7f70727fc640 1 -- 172.21.15.105:0/1746865453 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7078073d30 con 0x7f707c1023e0 2024-06-15T00:56:34.823 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:34.823 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:34.824 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7068065430 msgr2=0x7f70680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7068065430 0x7f70680678e0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 crypto rx=0x7f706c006fd0 tx=0x7f706c008040 comp rx=0 tx=0).stop 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 msgr2=0x7f707c1135c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 0x7f707c1135c0 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 crypto rx=0x7f70780048a0 tx=0x7f7078009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 shutdown_connections 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7068065430 0x7f70680678e0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f707c1023e0 0x7f707c1135c0 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 --2- 172.21.15.105:0/1746865453 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f707c0faa50 0x7f707c113080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 >> 172.21.15.105:0/1746865453 conn(0x7f707c070ab0 msgr2=0x7f707c0f9d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 shutdown_connections 2024-06-15T00:56:34.825 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:34.824+0000 7f7083fb2640 1 -- 172.21.15.105:0/1746865453 wait complete. 2024-06-15T00:56:35.376 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:35 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1746865453' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:35.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:35 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1746865453' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:36.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:36 smithi143 ceph-mon[35860]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:36.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:36 smithi105 ceph-mon[28877]: pgmap v244: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:38.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:38 smithi143 ceph-mon[35860]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:38.400 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:38.429 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:38 smithi105 ceph-mon[28877]: pgmap v245: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:38.671 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:40.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 -- 172.21.15.105:0/2766208492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd34102c90 msgr2=0x7fcd340fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:40.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/2766208492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd34102c90 0x7fcd340fa9e0 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 crypto rx=0x7fcd240048d0 tx=0x7fcd24009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 -- 172.21.15.105:0/2766208492 shutdown_connections 2024-06-15T00:56:40.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/2766208492 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd34102c90 0x7fcd340fa9e0 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/2766208492 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd341022f0 0x7fcd341026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 -- 172.21.15.105:0/2766208492 >> 172.21.15.105:0/2766208492 conn(0x7fcd34070a40 msgr2=0x7fcd34070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 -- 172.21.15.105:0/2766208492 shutdown_connections 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.097+0000 7fcd3b47d640 1 -- 172.21.15.105:0/2766208492 wait complete. 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd3b47d640 1 Processor -- start 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd3b47d640 1 -- start start 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd3b47d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd341022f0 0x7fcd34113030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd3b47d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 0x7fcd34113570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd3b47d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd34113b70 con 0x7fcd34102c90 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd3b47d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd3410cfb0 con 0x7fcd341022f0 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 0x7fcd34113570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 0x7fcd34113570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58744/0 (socket says 172.21.15.105:58744) 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 -- 172.21.15.105:0/553736508 learned_addr learned my addr 172.21.15.105:0/553736508 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:40.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd39219640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd341022f0 0x7fcd34113030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 -- 172.21.15.105:0/553736508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd341022f0 msgr2=0x7fcd34113030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd341022f0 0x7fcd34113030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 -- 172.21.15.105:0/553736508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd24005040 con 0x7fcd34102c90 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.098+0000 7fcd38a18640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 0x7fcd34113570 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcd24008fc0 tx=0x7fcd24009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.099+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd24038070 con 0x7fcd34102c90 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.099+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd3410d230 con 0x7fcd34102c90 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.099+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd2402cf00 con 0x7fcd34102c90 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.099+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fcd24009510 con 0x7fcd34102c90 2024-06-15T00:56:40.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.099+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd3410d6f0 con 0x7fcd34102c90 2024-06-15T00:56:40.102 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.100+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd34065510 con 0x7fcd34102c90 2024-06-15T00:56:40.102 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.100+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fcd24030020 con 0x7fcd34102c90 2024-06-15T00:56:40.102 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.101+0000 7fcd29ffb640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd20065430 0x7fcd200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:40.102 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.101+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fcd240a2100 con 0x7fcd34102c90 2024-06-15T00:56:40.103 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.101+0000 7fcd39219640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd20065430 0x7fcd200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:40.103 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.101+0000 7fcd39219640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd20065430 0x7fcd200678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd30009af0 tx=0x7fcd3000b040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:40.105 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.105+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fcd24070660 con 0x7fcd34102c90 2024-06-15T00:56:40.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:40 smithi105 ceph-mon[28877]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:40.278 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:40.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.302+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fcd34074960 con 0x7fcd34102c90 2024-06-15T00:56:40.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.302+0000 7fcd29ffb640 1 -- 172.21.15.105:0/553736508 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fcd24073c80 con 0x7fcd34102c90 2024-06-15T00:56:40.303 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:40.303 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd20065430 msgr2=0x7fcd200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd20065430 0x7fcd200678e0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 crypto rx=0x7fcd30009af0 tx=0x7fcd3000b040 comp rx=0 tx=0).stop 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 msgr2=0x7fcd34113570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 0x7fcd34113570 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 crypto rx=0x7fcd24008fc0 tx=0x7fcd24009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 shutdown_connections 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fcd20065430 0x7fcd200678e0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fcd34102c90 0x7fcd34113570 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.305 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 --2- 172.21.15.105:0/553736508 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fcd341022f0 0x7fcd34113030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:40.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 >> 172.21.15.105:0/553736508 conn(0x7fcd34070a40 msgr2=0x7fcd340f94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:40.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.304+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 shutdown_connections 2024-06-15T00:56:40.306 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:40.305+0000 7fcd3b47d640 1 -- 172.21.15.105:0/553736508 wait complete. 2024-06-15T00:56:40.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:40 smithi143 ceph-mon[35860]: pgmap v246: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:40.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:41.377 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:41 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/553736508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:41.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:41 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/553736508' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:42.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:42 smithi143 ceph-mon[35860]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:42.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:42 smithi105 ceph-mon[28877]: pgmap v247: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:43.884 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:44.148 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:44.249 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:44 smithi105 ceph-mon[28877]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:44.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:44 smithi143 ceph-mon[35860]: pgmap v248: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:45.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 -- 172.21.15.105:0/3909537450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 msgr2=0x7fdd6c101620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:45.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/3909537450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c101620 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 crypto rx=0x7fdd640048d0 tx=0x7fdd64009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 -- 172.21.15.105:0/3909537450 shutdown_connections 2024-06-15T00:56:45.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/3909537450 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd6c101b60 0x7fdd6c0fbad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/3909537450 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c101620 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.519 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 -- 172.21.15.105:0/3909537450 >> 172.21.15.105:0/3909537450 conn(0x7fdd6c0709e0 msgr2=0x7fdd6c070de0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 -- 172.21.15.105:0/3909537450 shutdown_connections 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 -- 172.21.15.105:0/3909537450 wait complete. 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 Processor -- start 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.517+0000 7fdd72fc2640 1 -- start start 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd72fc2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c10dfe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd72fc2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd6c101b60 0x7fdd6c10e520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd72fc2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c10ec00 con 0x7fdd6c101250 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd72fc2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd6c10ed70 con 0x7fdd6c101b60 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c10dfe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c10dfe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37012/0 (socket says 172.21.15.105:37012) 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 -- 172.21.15.105:0/540223891 learned_addr learned my addr 172.21.15.105:0/540223891 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:45.520 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 -- 172.21.15.105:0/540223891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd6c101b60 msgr2=0x7fdd6c10e520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd6c101b60 0x7fdd6c10e520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 -- 172.21.15.105:0/540223891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd64005040 con 0x7fdd6c101250 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd70d5e640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c10dfe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdd64004930 tx=0x7fdd64009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd64038070 con 0x7fdd6c101250 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd6c117050 con 0x7fdd6c101250 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdd64032d40 con 0x7fdd6c101250 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fdd6403dd70 con 0x7fdd6c101250 2024-06-15T00:56:45.521 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.518+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd6c1174c0 con 0x7fdd6c101250 2024-06-15T00:56:45.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.519+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd6c0fcae0 con 0x7fdd6c101250 2024-06-15T00:56:45.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.520+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fdd64030020 con 0x7fdd6c101250 2024-06-15T00:56:45.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.520+0000 7fdd697fa640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdd58065430 0x7fdd580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:45.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.520+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fdd640a3100 con 0x7fdd6c101250 2024-06-15T00:56:45.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.520+0000 7fdd6bfff640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdd58065430 0x7fdd580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:45.522 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.521+0000 7fdd6bfff640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdd58065430 0x7fdd580678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0fb3c0 tx=0x7fdd54006d20 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:45.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.524+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fdd640715b0 con 0x7fdd6c101250 2024-06-15T00:56:45.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.718+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fdd6c074980 con 0x7fdd6c101250 2024-06-15T00:56:45.719 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.719+0000 7fdd697fa640 1 -- 172.21.15.105:0/540223891 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fdd64074bd0 con 0x7fdd6c101250 2024-06-15T00:56:45.719 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:45.719 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:45.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdd58065430 msgr2=0x7fdd580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:45.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdd58065430 0x7fdd580678e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 crypto rx=0x7fdd6c0fb3c0 tx=0x7fdd54006d20 comp rx=0 tx=0).stop 2024-06-15T00:56:45.721 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 msgr2=0x7fdd6c10dfe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c10dfe0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 crypto rx=0x7fdd64004930 tx=0x7fdd64009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 shutdown_connections 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fdd58065430 0x7fdd580678e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fdd6c101b60 0x7fdd6c10e520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 --2- 172.21.15.105:0/540223891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fdd6c101250 0x7fdd6c10dfe0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 >> 172.21.15.105:0/540223891 conn(0x7fdd6c0709e0 msgr2=0x7fdd6c103fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 shutdown_connections 2024-06-15T00:56:45.722 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:45.721+0000 7fdd72fc2640 1 -- 172.21.15.105:0/540223891 wait complete. 2024-06-15T00:56:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:46 smithi105 ceph-mon[28877]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:46.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:46 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/540223891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:46.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:46 smithi143 ceph-mon[35860]: pgmap v249: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:46.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:46 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/540223891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:48.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:48 smithi143 ceph-mon[35860]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:48.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:48 smithi105 ceph-mon[28877]: pgmap v250: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:49.331 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:49.598 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:50.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:50 smithi105 ceph-mon[28877]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:50.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:50 smithi143 ceph-mon[35860]: pgmap v251: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:51.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 -- 172.21.15.105:0/1896547329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 msgr2=0x7ff1300fa9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:51.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 --2- 172.21.15.105:0/1896547329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff1300fa9e0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0048d0 tx=0x7ff11c009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.051 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 -- 172.21.15.105:0/1896547329 shutdown_connections 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 --2- 172.21.15.105:0/1896547329 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff1300fa9e0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 --2- 172.21.15.105:0/1896547329 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 0x7ff1301026c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 -- 172.21.15.105:0/1896547329 >> 172.21.15.105:0/1896547329 conn(0x7ff130070a40 msgr2=0x7ff130070e40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 -- 172.21.15.105:0/1896547329 shutdown_connections 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.049+0000 7ff136d40640 1 -- 172.21.15.105:0/1896547329 wait complete. 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff136d40640 1 Processor -- start 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff136d40640 1 -- start start 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff136d40640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 0x7ff130112fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff136d40640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff130113520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff136d40640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130113b20 con 0x7ff130102c90 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff136d40640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff13010cf60 con 0x7ff1301022f0 2024-06-15T00:56:51.052 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff130113520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff12ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff130113520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:37022/0 (socket says 172.21.15.105:37022) 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.050+0000 7ff12ffff640 1 -- 172.21.15.105:0/2787299415 learned_addr learned my addr 172.21.15.105:0/2787299415 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff134adc640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 0x7ff130112fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12ffff640 1 -- 172.21.15.105:0/2787299415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 msgr2=0x7ff130112fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12ffff640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 0x7ff130112fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12ffff640 1 -- 172.21.15.105:0/2787299415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff11c005040 con 0x7ff130102c90 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff134adc640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 0x7ff130112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12ffff640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff130113520 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0048a0 tx=0x7ff11c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11c038070 con 0x7ff130102c90 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff11c032d30 con 0x7ff130102c90 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff13010d1e0 con 0x7ff130102c90 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff11c009760 con 0x7ff130102c90 2024-06-15T00:56:51.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.051+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff13010d6a0 con 0x7ff130102c90 2024-06-15T00:56:51.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.052+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff130065510 con 0x7ff130102c90 2024-06-15T00:56:51.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.053+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff11c030020 con 0x7ff130102c90 2024-06-15T00:56:51.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.053+0000 7ff12d7fa640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff120065430 0x7ff1200678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:51.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.053+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff11c0a2b50 con 0x7ff130102c90 2024-06-15T00:56:51.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.053+0000 7ff134adc640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff120065430 0x7ff1200678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:51.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.053+0000 7ff134adc640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff120065430 0x7ff1200678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff130065260 tx=0x7ff124009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:51.057 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.057+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff11c071000 con 0x7ff130102c90 2024-06-15T00:56:51.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.254+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff130109310 con 0x7ff130102c90 2024-06-15T00:56:51.255 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.254+0000 7ff12d7fa640 1 -- 172.21.15.105:0/2787299415 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff11c074620 con 0x7ff130102c90 2024-06-15T00:56:51.255 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:51.255 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:51.257 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff120065430 msgr2=0x7ff1200678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff120065430 0x7ff1200678e0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 crypto rx=0x7ff130065260 tx=0x7ff124009040 comp rx=0 tx=0).stop 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 msgr2=0x7ff130113520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff130113520 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 crypto rx=0x7ff11c0048a0 tx=0x7ff11c009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 shutdown_connections 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff120065430 0x7ff1200678e0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff130102c90 0x7ff130113520 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.258 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 --2- 172.21.15.105:0/2787299415 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff1301022f0 0x7ff130112fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:51.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.256+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 >> 172.21.15.105:0/2787299415 conn(0x7ff130070a40 msgr2=0x7ff1300f94b0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:51.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.257+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 shutdown_connections 2024-06-15T00:56:51.259 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:51.257+0000 7ff136d40640 1 -- 172.21.15.105:0/2787299415 wait complete. 2024-06-15T00:56:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:52 smithi143 ceph-mon[35860]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:52 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2787299415' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:52.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:56:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:52 smithi105 ceph-mon[28877]: pgmap v252: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:52 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2787299415' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:52.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:56:53.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:56:53.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:56:53.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:56:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:56:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:56:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:56:54.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:54 smithi143 ceph-mon[35860]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:54.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:54 smithi105 ceph-mon[28877]: pgmap v253: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:54.946 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:56:55.208 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:56:55.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:55.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:56:56.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:56 smithi105 ceph-mon[28877]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:56.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:56 smithi143 ceph-mon[35860]: pgmap v254: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:56.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 -- 172.21.15.105:0/3729368048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 msgr2=0x7f84040fb4c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:56.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 --2- 172.21.15.105:0/3729368048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84040fb4c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 crypto rx=0x7f84000048d0 tx=0x7f8400009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.698 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 -- 172.21.15.105:0/3729368048 shutdown_connections 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 --2- 172.21.15.105:0/3729368048 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 0x7f84041000b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 --2- 172.21.15.105:0/3729368048 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84040fb4c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 -- 172.21.15.105:0/3729368048 >> 172.21.15.105:0/3729368048 conn(0x7f84040f6d90 msgr2=0x7f84040f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 -- 172.21.15.105:0/3729368048 shutdown_connections 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.696+0000 7f840c0d5640 1 -- 172.21.15.105:0/3729368048 wait complete. 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f840c0d5640 1 Processor -- start 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f840c0d5640 1 -- start start 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f840c0d5640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84041195e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f840c0d5640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 0x7f8404119b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f840c0d5640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f840411a220 con 0x7f84040fb0f0 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f840c0d5640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8404113580 con 0x7f84040fba90 2024-06-15T00:56:56.699 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84041195e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84041195e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58716/0 (socket says 172.21.15.105:58716) 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 -- 172.21.15.105:0/204236757 learned_addr learned my addr 172.21.15.105:0/204236757 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409670640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 0x7f8404119b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 -- 172.21.15.105:0/204236757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 msgr2=0x7f8404119b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 0x7f8404119b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 -- 172.21.15.105:0/204236757 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8400005040 con 0x7f84040fb0f0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409670640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 0x7f8404119b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.697+0000 7f8409e71640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84041195e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84000048a0 tx=0x7f8400009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.698+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8400038070 con 0x7f84040fb0f0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.698+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8400032d30 con 0x7f84040fb0f0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.698+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8404113800 con 0x7f84040fb0f0 2024-06-15T00:56:56.700 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.698+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8400009680 con 0x7f84040fb0f0 2024-06-15T00:56:56.701 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.698+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8404113cc0 con 0x7f84040fb0f0 2024-06-15T00:56:56.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.699+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84040660d0 con 0x7f84040fb0f0 2024-06-15T00:56:56.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.699+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8400030020 con 0x7f84040fb0f0 2024-06-15T00:56:56.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.699+0000 7f83fa7fc640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f83f0065430 0x7f83f00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:56:56.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.699+0000 7f8409670640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f83f0065430 0x7f83f00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:56:56.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.699+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f84000a2250 con 0x7f84040fb0f0 2024-06-15T00:56:56.702 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.700+0000 7f8409670640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f83f0065430 0x7f83f00678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f83f4006fd0 tx=0x7f83f4008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:56:56.704 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.704+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f84000707c0 con 0x7f84040fb0f0 2024-06-15T00:56:56.902 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.902+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f840404b480 con 0x7f84040fb0f0 2024-06-15T00:56:56.903 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.902+0000 7f83fa7fc640 1 -- 172.21.15.105:0/204236757 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8400073de0 con 0x7f84040fb0f0 2024-06-15T00:56:56.903 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:56:56.903 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f83f0065430 msgr2=0x7f83f00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f83f0065430 0x7f83f00678e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 crypto rx=0x7f83f4006fd0 tx=0x7f83f4008040 comp rx=0 tx=0).stop 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 msgr2=0x7f84041195e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84041195e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 crypto rx=0x7f84000048a0 tx=0x7f8400009da0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 shutdown_connections 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f83f0065430 0x7f83f00678e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f84040fba90 0x7f8404119b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 --2- 172.21.15.105:0/204236757 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f84040fb0f0 0x7f84041195e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 >> 172.21.15.105:0/204236757 conn(0x7f84040f6d90 msgr2=0x7f84040f8450 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:56:56.905 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 shutdown_connections 2024-06-15T00:56:56.906 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:56:56.904+0000 7f840c0d5640 1 -- 172.21.15.105:0/204236757 wait complete. 2024-06-15T00:56:57.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:57 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/204236757' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:57.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:57 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/204236757' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:56:58.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:56:58 smithi143 ceph-mon[35860]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:56:58.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:56:58 smithi105 ceph-mon[28877]: pgmap v255: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:00.386 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:00 smithi143 ceph-mon[35860]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:00.507 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:00.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:00 smithi105 ceph-mon[28877]: pgmap v256: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:00.773 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:02.235 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 -- 172.21.15.105:0/4208057330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 msgr2=0x7fa95c06fb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 --2- 172.21.15.105:0/4208057330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c06fb80 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 crypto rx=0x7fa94c0048d0 tx=0x7fa94c009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 -- 172.21.15.105:0/4208057330 shutdown_connections 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 --2- 172.21.15.105:0/4208057330 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95c0700c0 0x7fa95c108630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 --2- 172.21.15.105:0/4208057330 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c06fb80 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 -- 172.21.15.105:0/4208057330 >> 172.21.15.105:0/4208057330 conn(0x7fa95c0f7e50 msgr2=0x7fa95c0fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 -- 172.21.15.105:0/4208057330 shutdown_connections 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.217+0000 7fa961014640 1 -- 172.21.15.105:0/4208057330 wait complete. 2024-06-15T00:57:02.236 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa961014640 1 Processor -- start 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa961014640 1 -- start start 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa961014640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c115150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa961014640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95c0700c0 0x7fa95c115690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa961014640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c115c90 con 0x7fa95c06f7b0 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa961014640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa95c10f0d0 con 0x7fa95c0700c0 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa95ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c115150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa95ad9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c115150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:58738/0 (socket says 172.21.15.105:58738) 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.218+0000 7fa95ad9d640 1 -- 172.21.15.105:0/822497613 learned_addr learned my addr 172.21.15.105:0/822497613 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa95259c640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95c0700c0 0x7fa95c115690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa95ad9d640 1 -- 172.21.15.105:0/822497613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95c0700c0 msgr2=0x7fa95c115690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa95ad9d640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95c0700c0 0x7fa95c115690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.237 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa95ad9d640 1 -- 172.21.15.105:0/822497613 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c005040 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa95ad9d640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c115150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa94c000c00 tx=0x7fa94c009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa94c038070 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa95c10f350 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa94c0096f0 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fa94c032da0 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.219+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa95c10f810 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.220+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa95c070ef0 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.220+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fa94c030020 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.221+0000 7fa953fff640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa948065430 0x7fa9480678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.221+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fa94c0a2ab0 con 0x7fa95c06f7b0 2024-06-15T00:57:02.238 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.221+0000 7fa95259c640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa948065430 0x7fa9480678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:02.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.221+0000 7fa95259c640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa948065430 0x7fa9480678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa944007fd0 tx=0x7fa944009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:02.239 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.225+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fa94c070f60 con 0x7fa95c06f7b0 2024-06-15T00:57:02.385 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:02 smithi143 ceph-mon[35860]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:02.421 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.421+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fa95c10b500 con 0x7fa95c06f7b0 2024-06-15T00:57:02.422 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:02 smithi105 ceph-mon[28877]: pgmap v257: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:02.422 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.421+0000 7fa953fff640 1 -- 172.21.15.105:0/822497613 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fa94c074580 con 0x7fa95c06f7b0 2024-06-15T00:57:02.422 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:02.422 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.423+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa948065430 msgr2=0x7fa9480678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.423+0000 7fa961014640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa948065430 0x7fa9480678e0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 crypto rx=0x7fa944007fd0 tx=0x7fa944009040 comp rx=0 tx=0).stop 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.423+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 msgr2=0x7fa95c115150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.423+0000 7fa961014640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c115150 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 crypto rx=0x7fa94c000c00 tx=0x7fa94c009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 shutdown_connections 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fa948065430 0x7fa9480678e0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.424 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fa95c0700c0 0x7fa95c115690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 --2- 172.21.15.105:0/822497613 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fa95c06f7b0 0x7fa95c115150 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:02.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 >> 172.21.15.105:0/822497613 conn(0x7fa95c0f7e50 msgr2=0x7fa95c0f9370 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:02.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 shutdown_connections 2024-06-15T00:57:02.425 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:02.424+0000 7fa961014640 1 -- 172.21.15.105:0/822497613 wait complete. 2024-06-15T00:57:03.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:03 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/822497613' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:03.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:03 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/822497613' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:04.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:04 smithi105 ceph-mon[28877]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:04.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:04 smithi143 ceph-mon[35860]: pgmap v258: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:05.955 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:06.219 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:06.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:06 smithi105 ceph-mon[28877]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:06.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:06 smithi143 ceph-mon[35860]: pgmap v259: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:07.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 -- 172.21.15.105:0/777835459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d041011a0 msgr2=0x7f8d04101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:07.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 --2- 172.21.15.105:0/777835459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d041011a0 0x7f8d04101570 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 crypto rx=0x7f8cf40048d0 tx=0x7f8cf4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 -- 172.21.15.105:0/777835459 shutdown_connections 2024-06-15T00:57:07.727 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 --2- 172.21.15.105:0/777835459 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d04101b40 0x7f8d040fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 --2- 172.21.15.105:0/777835459 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d041011a0 0x7f8d04101570 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 -- 172.21.15.105:0/777835459 >> 172.21.15.105:0/777835459 conn(0x7f8d040f6d90 msgr2=0x7f8d040f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 -- 172.21.15.105:0/777835459 shutdown_connections 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 -- 172.21.15.105:0/777835459 wait complete. 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 Processor -- start 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.725+0000 7f8d0a026640 1 -- start start 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d0a026640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d041011a0 0x7f8d0410ded0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d0a026640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 0x7f8d0410e410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d0a026640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0410ea60 con 0x7f8d04101b40 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d0a026640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0410ebd0 con 0x7f8d041011a0 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 0x7f8d0410e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 0x7f8d0410e410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:49700/0 (socket says 172.21.15.105:49700) 2024-06-15T00:57:07.728 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 -- 172.21.15.105:0/3408318862 learned_addr learned my addr 172.21.15.105:0/3408318862 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d037fe640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d041011a0 0x7f8d0410ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 -- 172.21.15.105:0/3408318862 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d041011a0 msgr2=0x7f8d0410ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d041011a0 0x7f8d0410ded0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 -- 172.21.15.105:0/3408318862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cf4005040 con 0x7f8d04101b40 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8cfbfff640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 0x7f8d0410e410 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00c790 tx=0x7f8cec0043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cec00a820 con 0x7f8d04101b40 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cec00ae30 con 0x7f8d04101b40 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d04116f40 con 0x7f8d04101b40 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f8cec0113a0 con 0x7f8d04101b40 2024-06-15T00:57:07.729 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.726+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d04117450 con 0x7f8d04101b40 2024-06-15T00:57:07.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.727+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d040fc910 con 0x7f8d04101b40 2024-06-15T00:57:07.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.728+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f8cec012400 con 0x7f8d04101b40 2024-06-15T00:57:07.730 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.728+0000 7f8d00ff9640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cf0065430 0x7f8cf00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:07.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.728+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f8cec082620 con 0x7f8d04101b40 2024-06-15T00:57:07.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.728+0000 7f8d037fe640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cf0065430 0x7f8cf00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:07.731 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.729+0000 7f8d037fe640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cf0065430 0x7f8cf00678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4003080 tx=0x7f8cf4009da0 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:07.733 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.732+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f8cec050c10 con 0x7f8d04101b40 2024-06-15T00:57:07.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.927+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f8d040fa990 con 0x7f8d04101b40 2024-06-15T00:57:07.928 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.927+0000 7f8d00ff9640 1 -- 172.21.15.105:0/3408318862 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f8cec054230 con 0x7f8d04101b40 2024-06-15T00:57:07.928 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:07.928 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cf0065430 msgr2=0x7f8cf00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cf0065430 0x7f8cf00678e0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 crypto rx=0x7f8cf4003080 tx=0x7f8cf4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 msgr2=0x7f8d0410e410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 0x7f8d0410e410 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 crypto rx=0x7f8cec00c790 tx=0x7f8cec0043c0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 shutdown_connections 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8cf0065430 0x7f8cf00678e0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.930 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f8d04101b40 0x7f8d0410e410 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 --2- 172.21.15.105:0/3408318862 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f8d041011a0 0x7f8d0410ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:07.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.929+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 >> 172.21.15.105:0/3408318862 conn(0x7f8d040f6d90 msgr2=0x7f8d040f7d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:07.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.930+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 shutdown_connections 2024-06-15T00:57:07.931 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:07.930+0000 7f8d0a026640 1 -- 172.21.15.105:0/3408318862 wait complete. 2024-06-15T00:57:08.276 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:08 smithi105 ceph-mon[28877]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:08.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:08 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3408318862' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:08.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:08 smithi143 ceph-mon[35860]: pgmap v260: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:08.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:08 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3408318862' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:10.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:10 smithi105 ceph-mon[28877]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:10.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:10.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:10 smithi143 ceph-mon[35860]: pgmap v261: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:10.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:11.569 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:11.832 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:12.239 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:12 smithi105 ceph-mon[28877]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:12.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:12 smithi143 ceph-mon[35860]: pgmap v262: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:13.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 -- 172.21.15.105:0/3357723539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4758101b70 msgr2=0x7f47580fb200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:13.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 --2- 172.21.15.105:0/3357723539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4758101b70 0x7f47580fb200 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 crypto rx=0x7f474c0048d0 tx=0x7f474c009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 -- 172.21.15.105:0/3357723539 shutdown_connections 2024-06-15T00:57:13.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 --2- 172.21.15.105:0/3357723539 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4758101b70 0x7f47580fb200 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 --2- 172.21.15.105:0/3357723539 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f47581011d0 0x7f47581015a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.342 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 -- 172.21.15.105:0/3357723539 >> 172.21.15.105:0/3357723539 conn(0x7f47580f6dc0 msgr2=0x7f47580f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 -- 172.21.15.105:0/3357723539 shutdown_connections 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.339+0000 7f475d1d2640 1 -- 172.21.15.105:0/3357723539 wait complete. 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f475d1d2640 1 Processor -- start 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f475d1d2640 1 -- start start 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f475d1d2640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 0x7f4758112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f475d1d2640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4758101b70 0x7f47581134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f475d1d2640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4758113b80 con 0x7f47581011d0 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f475d1d2640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f475810cf70 con 0x7f4758101b70 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f4756d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 0x7f4758112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f4756d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 0x7f4758112fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52966/0 (socket says 172.21.15.105:52966) 2024-06-15T00:57:13.343 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.340+0000 7f4756d9d640 1 -- 172.21.15.105:0/2708248007 learned_addr learned my addr 172.21.15.105:0/2708248007 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f475659c640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4758101b70 0x7f47581134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f4756d9d640 1 -- 172.21.15.105:0/2708248007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4758101b70 msgr2=0x7f47581134f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f4756d9d640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4758101b70 0x7f47581134f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f4756d9d640 1 -- 172.21.15.105:0/2708248007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474c005040 con 0x7f47581011d0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f4756d9d640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 0x7f4758112fb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f474800b790 tx=0x7f47480091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f474800c700 con 0x7f47581011d0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f474800cd10 con 0x7f47581011d0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f475810d250 con 0x7f47581011d0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4748013370 con 0x7f47581011d0 2024-06-15T00:57:13.344 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.341+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475810d770 con 0x7f47581011d0 2024-06-15T00:57:13.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.342+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47580fc940 con 0x7f47581011d0 2024-06-15T00:57:13.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.343+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4748012070 con 0x7f47581011d0 2024-06-15T00:57:13.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.343+0000 7f473f7fe640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4740065430 0x7f47400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:13.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.343+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4748081ec0 con 0x7f47581011d0 2024-06-15T00:57:13.345 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.343+0000 7f475659c640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4740065430 0x7f47400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:13.346 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.344+0000 7f475659c640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4740065430 0x7f47400678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f474c004540 tx=0x7f474c035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:13.347 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.347+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4748050370 con 0x7f47581011d0 2024-06-15T00:57:13.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.546+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f475804b4b0 con 0x7f47581011d0 2024-06-15T00:57:13.547 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.547+0000 7f473f7fe640 1 -- 172.21.15.105:0/2708248007 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4748053990 con 0x7f47581011d0 2024-06-15T00:57:13.547 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:13.548 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:13.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4740065430 msgr2=0x7f47400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:13.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4740065430 0x7f47400678e0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 crypto rx=0x7f474c004540 tx=0x7f474c035040 comp rx=0 tx=0).stop 2024-06-15T00:57:13.549 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 msgr2=0x7f4758112fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 0x7f4758112fb0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 crypto rx=0x7f474800b790 tx=0x7f47480091d0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 shutdown_connections 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4740065430 0x7f47400678e0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4758101b70 0x7f47581134f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 --2- 172.21.15.105:0/2708248007 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f47581011d0 0x7f4758112fb0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 >> 172.21.15.105:0/2708248007 conn(0x7f47580f6dc0 msgr2=0x7f47580657d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 shutdown_connections 2024-06-15T00:57:13.550 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:13.549+0000 7f475d1d2640 1 -- 172.21.15.105:0/2708248007 wait complete. 2024-06-15T00:57:14.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:14 smithi105 ceph-mon[28877]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:14.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:14 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2708248007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:14.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:14 smithi143 ceph-mon[35860]: pgmap v263: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:14.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:14 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2708248007' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:16.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:16 smithi105 ceph-mon[28877]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:16.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:16 smithi143 ceph-mon[35860]: pgmap v264: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:17.099 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:17.371 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:18.453 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:18 smithi105 ceph-mon[28877]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:18.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:18 smithi143 ceph-mon[35860]: pgmap v265: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:18.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 -- 172.21.15.105:0/2227513236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb90101010 msgr2=0x7fbb901013e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:18.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2227513236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb90101010 0x7fbb901013e0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c0048d0 tx=0x7fbb8c009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:18.810 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 -- 172.21.15.105:0/2227513236 shutdown_connections 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2227513236 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb901019b0 0x7fbb900fb040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2227513236 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb90101010 0x7fbb901013e0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 -- 172.21.15.105:0/2227513236 >> 172.21.15.105:0/2227513236 conn(0x7fbb900f6ba0 msgr2=0x7fbb900f8fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 -- 172.21.15.105:0/2227513236 shutdown_connections 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.803+0000 7fbb9639d640 1 -- 172.21.15.105:0/2227513236 wait complete. 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb9639d640 1 Processor -- start 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb9639d640 1 -- start start 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb9639d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90101010 0x7fbb9010dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb9639d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 0x7fbb9010e250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb9639d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9010e8a0 con 0x7fbb901019b0 2024-06-15T00:57:18.811 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb9639d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb9010ea10 con 0x7fbb90101010 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb94b9a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 0x7fbb9010e250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb94b9a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 0x7fbb9010e250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52990/0 (socket says 172.21.15.105:52990) 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.804+0000 7fbb94b9a640 1 -- 172.21.15.105:0/2257548136 learned_addr learned my addr 172.21.15.105:0/2257548136 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb9539b640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90101010 0x7fbb9010dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb94b9a640 1 -- 172.21.15.105:0/2257548136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90101010 msgr2=0x7fbb9010dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb94b9a640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90101010 0x7fbb9010dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb94b9a640 1 -- 172.21.15.105:0/2257548136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb8c005040 con 0x7fbb901019b0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb94b9a640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 0x7fbb9010e250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c810 tx=0x7fbb800043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb8000a820 con 0x7fbb901019b0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb8000ae30 con 0x7fbb901019b0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fbb800115c0 con 0x7fbb901019b0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb90116d80 con 0x7fbb901019b0 2024-06-15T00:57:18.812 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.805+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb90117290 con 0x7fbb901019b0 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.806+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb900fc780 con 0x7fbb901019b0 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.807+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fbb80012420 con 0x7fbb901019b0 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.807+0000 7fbb85ffb640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbb7c065430 0x7fbb7c0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.807+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fbb80082930 con 0x7fbb901019b0 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.807+0000 7fbb9539b640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbb7c065430 0x7fbb7c0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.808+0000 7fbb9539b640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbb7c065430 0x7fbb7c0678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c004540 tx=0x7fbb8c035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:18.813 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:18.811+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fbb80050e60 con 0x7fbb901019b0 2024-06-15T00:57:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.010+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fbb900fa800 con 0x7fbb901019b0 2024-06-15T00:57:19.011 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.010+0000 7fbb85ffb640 1 -- 172.21.15.105:0/2257548136 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fbb800126d0 con 0x7fbb901019b0 2024-06-15T00:57:19.011 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:19.011 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbb7c065430 msgr2=0x7fbb7c0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbb7c065430 0x7fbb7c0678e0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 crypto rx=0x7fbb8c004540 tx=0x7fbb8c035040 comp rx=0 tx=0).stop 2024-06-15T00:57:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 msgr2=0x7fbb9010e250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 0x7fbb9010e250 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 crypto rx=0x7fbb8000c810 tx=0x7fbb800043c0 comp rx=0 tx=0).stop 2024-06-15T00:57:19.013 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 shutdown_connections 2024-06-15T00:57:19.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fbb7c065430 0x7fbb7c0678e0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:19.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fbb901019b0 0x7fbb9010e250 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:19.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 --2- 172.21.15.105:0/2257548136 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fbb90101010 0x7fbb9010dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:19.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 >> 172.21.15.105:0/2257548136 conn(0x7fbb900f6ba0 msgr2=0x7fbb900f7c30 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:19.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 shutdown_connections 2024-06-15T00:57:19.014 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:19.013+0000 7fbb9639d640 1 -- 172.21.15.105:0/2257548136 wait complete. 2024-06-15T00:57:19.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:19 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2257548136' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:19.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:19 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2257548136' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:20.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:20 smithi105 ceph-mon[28877]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:20.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:20 smithi143 ceph-mon[35860]: pgmap v266: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:22.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:22 smithi105 ceph-mon[28877]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:22.548 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:22.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:22 smithi143 ceph-mon[35860]: pgmap v267: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:22.816 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:24.316 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 -- 172.21.15.105:0/83657725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fd2d0 msgr2=0x7fb3ec0fd6a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:24.316 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/83657725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec0fd6a0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4009c50 tx=0x7fb3e4004d10 comp rx=0 tx=0).stop 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 -- 172.21.15.105:0/83657725 shutdown_connections 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/83657725 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1062b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/83657725 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec0fd6a0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 -- 172.21.15.105:0/83657725 >> 172.21.15.105:0/83657725 conn(0x7fb3ec0f6d20 msgr2=0x7fb3ec0f9130 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 -- 172.21.15.105:0/83657725 shutdown_connections 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.301+0000 7fb3f37ef640 1 -- 172.21.15.105:0/83657725 wait complete. 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f37ef640 1 Processor -- start 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f37ef640 1 -- start start 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f37ef640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec112f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f37ef640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1134d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f37ef640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec113ad0 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f37ef640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3ec10cf10 con 0x7fb3ec0fd2d0 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1134d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:24.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1134d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45100/0 (socket says 172.21.15.105:45100) 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 -- 172.21.15.105:0/1592397440 learned_addr learned my addr 172.21.15.105:0/1592397440 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f158b640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec112f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 -- 172.21.15.105:0/1592397440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fd2d0 msgr2=0x7fb3ec112f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec112f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 -- 172.21.15.105:0/1592397440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40098b0 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f158b640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec112f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.302+0000 7fb3f0d8a640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1134d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c810 tx=0x7fb3e00043c0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.303+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3e000a820 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.303+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3ec10d1f0 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.303+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3e000ae30 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.303+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fb3e00115c0 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.303+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3ec10d710 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.304+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3ec066130 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.304+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fb3e0012420 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.304+0000 7fb3e9ffb640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb3d4065430 0x7fb3d40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.304+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fb3e0082ab0 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.304+0000 7fb3f158b640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb3d4065430 0x7fb3d40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.305+0000 7fb3f158b640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb3d4065430 0x7fb3d40678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4000f80 tx=0x7fb3e4004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:24.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.308+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fb3e0050f60 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:24 smithi105 ceph-mon[28877]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:24.510 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.510+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb3ec10e600 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.511 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.510+0000 7fb3e9ffb640 1 -- 172.21.15.105:0/1592397440 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fb3e00126d0 con 0x7fb3ec0fdc70 2024-06-15T00:57:24.511 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:24.511 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.512+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb3d4065430 msgr2=0x7fb3d40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.512+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb3d4065430 0x7fb3d40678e0 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 crypto rx=0x7fb3e4000f80 tx=0x7fb3e4004d10 comp rx=0 tx=0).stop 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.512+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 msgr2=0x7fb3ec1134d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.512+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1134d0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 crypto rx=0x7fb3e000c810 tx=0x7fb3e00043c0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.512+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 shutdown_connections 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.513+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fb3d4065430 0x7fb3d40678e0 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.513+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fb3ec0fdc70 0x7fb3ec1134d0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.513 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.513+0000 7fb3f37ef640 1 --2- 172.21.15.105:0/1592397440 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fb3ec0fd2d0 0x7fb3ec112f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:24.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.513+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 >> 172.21.15.105:0/1592397440 conn(0x7fb3ec0f6d20 msgr2=0x7fb3ec0f8280 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:24.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.513+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 shutdown_connections 2024-06-15T00:57:24.514 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:24.513+0000 7fb3f37ef640 1 -- 172.21.15.105:0/1592397440 wait complete. 2024-06-15T00:57:24.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:24 smithi143 ceph-mon[35860]: pgmap v268: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:25.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:25 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1592397440' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:25.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:25.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:25 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1592397440' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:25.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:26.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:26 smithi105 ceph-mon[28877]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:26.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:26 smithi143 ceph-mon[35860]: pgmap v269: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:28.113 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:28.377 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:28.488 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:28 smithi105 ceph-mon[28877]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:28.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:28 smithi143 ceph-mon[35860]: pgmap v270: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:29.850 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 -- 172.21.15.105:0/4130244362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c071440 msgr2=0x7feb2c06f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 --2- 172.21.15.105:0/4130244362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c071440 0x7feb2c06f810 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 crypto rx=0x7feb200048d0 tx=0x7feb20009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 -- 172.21.15.105:0/4130244362 shutdown_connections 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 --2- 172.21.15.105:0/4130244362 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c06fd50 0x7feb2c0701c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 --2- 172.21.15.105:0/4130244362 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c071440 0x7feb2c06f810 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 -- 172.21.15.105:0/4130244362 >> 172.21.15.105:0/4130244362 conn(0x7feb2c0f7e80 msgr2=0x7feb2c0fa290 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 -- 172.21.15.105:0/4130244362 shutdown_connections 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.848+0000 7feb3315e640 1 -- 172.21.15.105:0/4130244362 wait complete. 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb3315e640 1 Processor -- start 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb3315e640 1 -- start start 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb3315e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 0x7feb2c1195d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:29.851 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb3315e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c071440 0x7feb2c119b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb3315e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c11a180 con 0x7feb2c06fd50 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb3315e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2c113570 con 0x7feb2c071440 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb30efa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 0x7feb2c1195d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb30efa640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 0x7feb2c1195d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:45114/0 (socket says 172.21.15.105:45114) 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb30efa640 1 -- 172.21.15.105:0/3969380587 learned_addr learned my addr 172.21.15.105:0/3969380587 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb2bfff640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c071440 0x7feb2c119b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb30efa640 1 -- 172.21.15.105:0/3969380587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c071440 msgr2=0x7feb2c119b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb30efa640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c071440 0x7feb2c119b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb30efa640 1 -- 172.21.15.105:0/3969380587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb20005040 con 0x7feb2c06fd50 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.849+0000 7feb2bfff640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c071440 0x7feb2c119b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.850+0000 7feb30efa640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 0x7feb2c1195d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7feb20032410 tx=0x7feb20009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.850+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb20038070 con 0x7feb2c06fd50 2024-06-15T00:57:29.852 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.850+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2c1137f0 con 0x7feb2c06fd50 2024-06-15T00:57:29.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.850+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb20032df0 con 0x7feb2c06fd50 2024-06-15T00:57:29.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.850+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7feb2003de30 con 0x7feb2c06fd50 2024-06-15T00:57:29.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.850+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2c113cb0 con 0x7feb2c06fd50 2024-06-15T00:57:29.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.851+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2c070f50 con 0x7feb2c06fd50 2024-06-15T00:57:29.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.851+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7feb20030020 con 0x7feb2c06fd50 2024-06-15T00:57:29.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.851+0000 7feb297fa640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7feb1c0653d0 0x7feb1c067880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:29.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.852+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7feb200a30f0 con 0x7feb2c06fd50 2024-06-15T00:57:29.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.852+0000 7feb2bfff640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7feb1c0653d0 0x7feb1c067880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:29.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.852+0000 7feb2bfff640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7feb1c0653d0 0x7feb1c067880 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feb2c114e90 tx=0x7feb18009040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:29.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:29.856+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7feb20071650 con 0x7feb2c06fd50 2024-06-15T00:57:30.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.052+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7feb2c04b480 con 0x7feb2c06fd50 2024-06-15T00:57:30.053 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.053+0000 7feb297fa640 1 -- 172.21.15.105:0/3969380587 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7feb200a1e60 con 0x7feb2c06fd50 2024-06-15T00:57:30.053 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:30.053 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7feb1c0653d0 msgr2=0x7feb1c067880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7feb1c0653d0 0x7feb1c067880 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 crypto rx=0x7feb2c114e90 tx=0x7feb18009040 comp rx=0 tx=0).stop 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 msgr2=0x7feb2c1195d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 0x7feb2c1195d0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 crypto rx=0x7feb20032410 tx=0x7feb20009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 shutdown_connections 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7feb1c0653d0 0x7feb1c067880 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:30.055 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7feb2c071440 0x7feb2c119b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:30.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 --2- 172.21.15.105:0/3969380587 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7feb2c06fd50 0x7feb2c1195d0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:30.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 >> 172.21.15.105:0/3969380587 conn(0x7feb2c0f7e80 msgr2=0x7feb2c0f90a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:30.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 shutdown_connections 2024-06-15T00:57:30.056 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:30.055+0000 7feb3315e640 1 -- 172.21.15.105:0/3969380587 wait complete. 2024-06-15T00:57:30.491 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:30 smithi105 ceph-mon[28877]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:30.491 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:30 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3969380587' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:30.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:30 smithi143 ceph-mon[35860]: pgmap v271: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:30.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:30 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3969380587' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:32.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:32 smithi105 ceph-mon[28877]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:32.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:32 smithi143 ceph-mon[35860]: pgmap v272: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:33.651 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:33.915 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:34.354 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:34 smithi105 ceph-mon[28877]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:34.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:34 smithi143 ceph-mon[35860]: pgmap v273: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:35.316 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 -- 172.21.15.105:0/4284958566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf81022c0 msgr2=0x7f4cf8102690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 --2- 172.21.15.105:0/4284958566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf81022c0 0x7f4cf8102690 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 crypto rx=0x7f4cf40048d0 tx=0x7f4cf4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 -- 172.21.15.105:0/4284958566 shutdown_connections 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 --2- 172.21.15.105:0/4284958566 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf8102c60 0x7f4cf80fc2f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 --2- 172.21.15.105:0/4284958566 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf81022c0 0x7f4cf8102690 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 -- 172.21.15.105:0/4284958566 >> 172.21.15.105:0/4284958566 conn(0x7f4cf80f7e50 msgr2=0x7f4cf80fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 -- 172.21.15.105:0/4284958566 shutdown_connections 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.314+0000 7f4d003ea640 1 -- 172.21.15.105:0/4284958566 wait complete. 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4d003ea640 1 Processor -- start 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4d003ea640 1 -- start start 2024-06-15T00:57:35.317 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4d003ea640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81022c0 0x7f4cf806f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4d003ea640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 0x7f4cf806fcb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4d003ea640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf80701f0 con 0x7f4cf8102c60 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4d003ea640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cf8070360 con 0x7f4cf81022c0 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfd985640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 0x7f4cf806fcb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfd985640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 0x7f4cf806fcb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33584/0 (socket says 172.21.15.105:33584) 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfd985640 1 -- 172.21.15.105:0/3790550014 learned_addr learned my addr 172.21.15.105:0/3790550014 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfe186640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81022c0 0x7f4cf806f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfd985640 1 -- 172.21.15.105:0/3790550014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81022c0 msgr2=0x7f4cf806f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfd985640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81022c0 0x7f4cf806f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfd985640 1 -- 172.21.15.105:0/3790550014 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf4005040 con 0x7f4cf8102c60 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.315+0000 7f4cfe186640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81022c0 0x7f4cf806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.316+0000 7f4cfd985640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 0x7f4cf806fcb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c830 tx=0x7f4ce8004430 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:35.318 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.316+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce800a850 con 0x7f4cf8102c60 2024-06-15T00:57:35.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.316+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ce800ae60 con 0x7f4cf8102c60 2024-06-15T00:57:35.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.316+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f4ce80113a0 con 0x7f4cf8102c60 2024-06-15T00:57:35.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.316+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cf80705f0 con 0x7f4cf8102c60 2024-06-15T00:57:35.319 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.316+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cf8119970 con 0x7f4cf8102c60 2024-06-15T00:57:35.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.317+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cf80fda30 con 0x7f4cf8102c60 2024-06-15T00:57:35.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.317+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f4ce8012400 con 0x7f4cf8102c60 2024-06-15T00:57:35.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.318+0000 7f4ceeffd640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4ce4065430 0x7f4ce40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:35.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.318+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f4ce80826e0 con 0x7f4cf8102c60 2024-06-15T00:57:35.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.318+0000 7f4cfe186640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4ce4065430 0x7f4ce40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:35.320 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.318+0000 7f4cfe186640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4ce4065430 0x7f4ce40678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4007ac0 tx=0x7f4cf4035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:35.322 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.322+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f4ce8050cd0 con 0x7f4cf8102c60 2024-06-15T00:57:35.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.522+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f4cf8070e60 con 0x7f4cf8102c60 2024-06-15T00:57:35.523 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.523+0000 7f4ceeffd640 1 -- 172.21.15.105:0/3790550014 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f4ce80542f0 con 0x7f4cf8102c60 2024-06-15T00:57:35.524 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:35.524 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:35.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4ce4065430 msgr2=0x7f4ce40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:35.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4ce4065430 0x7f4ce40678e0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 crypto rx=0x7f4cf4007ac0 tx=0x7f4cf4035040 comp rx=0 tx=0).stop 2024-06-15T00:57:35.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 msgr2=0x7f4cf806fcb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:35.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 0x7f4cf806fcb0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 crypto rx=0x7f4ce800c830 tx=0x7f4ce8004430 comp rx=0 tx=0).stop 2024-06-15T00:57:35.525 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 shutdown_connections 2024-06-15T00:57:35.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f4ce4065430 0x7f4ce40678e0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f4cf8102c60 0x7f4cf806fcb0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 --2- 172.21.15.105:0/3790550014 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f4cf81022c0 0x7f4cf806f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:35.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 >> 172.21.15.105:0/3790550014 conn(0x7f4cf80f7e50 msgr2=0x7f4cf80f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:35.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 shutdown_connections 2024-06-15T00:57:35.526 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:35.525+0000 7f4d003ea640 1 -- 172.21.15.105:0/3790550014 wait complete. 2024-06-15T00:57:36.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:36 smithi105 ceph-mon[28877]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:36.529 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:36 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3790550014' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:36.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:36 smithi143 ceph-mon[35860]: pgmap v274: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:36.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:36 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3790550014' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:38.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:38 smithi105 ceph-mon[28877]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:38.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:38 smithi143 ceph-mon[35860]: pgmap v275: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:39.133 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:39.397 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:40.456 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:40 smithi105 ceph-mon[28877]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:40.456 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:40.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:40 smithi143 ceph-mon[35860]: pgmap v276: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:40.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:40.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 -- 172.21.15.105:0/409621785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3101011d0 msgr2=0x7ff3101015a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:40.961 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 --2- 172.21.15.105:0/409621785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3101011d0 0x7ff3101015a0 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 crypto rx=0x7ff30c0048d0 tx=0x7ff30c009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 -- 172.21.15.105:0/409621785 shutdown_connections 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 --2- 172.21.15.105:0/409621785 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff310101b70 0x7ff3100fb200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 --2- 172.21.15.105:0/409621785 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff3101011d0 0x7ff3101015a0 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 -- 172.21.15.105:0/409621785 >> 172.21.15.105:0/409621785 conn(0x7ff3100f6d80 msgr2=0x7ff3100f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 -- 172.21.15.105:0/409621785 shutdown_connections 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.959+0000 7ff317a8d640 1 -- 172.21.15.105:0/409621785 wait complete. 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff317a8d640 1 Processor -- start 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff317a8d640 1 -- start start 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff317a8d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3101011d0 0x7ff310102050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff317a8d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 0x7ff310102590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:40.962 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff317a8d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff310102b60 con 0x7ff310101b70 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff317a8d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff310102cd0 con 0x7ff3101011d0 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff315028640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 0x7ff310102590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff315028640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 0x7ff310102590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33606/0 (socket says 172.21.15.105:33606) 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff315028640 1 -- 172.21.15.105:0/2169303028 learned_addr learned my addr 172.21.15.105:0/2169303028 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.960+0000 7ff315829640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3101011d0 0x7ff310102050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff315028640 1 -- 172.21.15.105:0/2169303028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3101011d0 msgr2=0x7ff310102050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff315028640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3101011d0 0x7ff310102050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff315028640 1 -- 172.21.15.105:0/2169303028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c005040 con 0x7ff310101b70 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff315028640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 0x7ff310102590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff30000b790 tx=0x7ff3000091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff30000c700 con 0x7ff310101b70 2024-06-15T00:57:40.963 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff30000cd10 con 0x7ff310101b70 2024-06-15T00:57:40.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3101192a0 con 0x7ff310101b70 2024-06-15T00:57:40.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7ff300013370 con 0x7ff310101b70 2024-06-15T00:57:40.964 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.961+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3101196e0 con 0x7ff310101b70 2024-06-15T00:57:40.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.962+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3100fc940 con 0x7ff310101b70 2024-06-15T00:57:40.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.962+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7ff300012070 con 0x7ff310101b70 2024-06-15T00:57:40.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.963+0000 7ff3067fc640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff2fc065430 0x7ff2fc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:40.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.963+0000 7ff315829640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff2fc065430 0x7ff2fc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:40.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.963+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7ff300081ca0 con 0x7ff310101b70 2024-06-15T00:57:40.965 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.963+0000 7ff315829640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff2fc065430 0x7ff2fc0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff30c02a600 tx=0x7ff30c035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:40.967 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:40.967+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7ff300086070 con 0x7ff310101b70 2024-06-15T00:57:41.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.166+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7ff310102e40 con 0x7ff310101b70 2024-06-15T00:57:41.167 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.167+0000 7ff3067fc640 1 -- 172.21.15.105:0/2169303028 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7ff310102e40 con 0x7ff310101b70 2024-06-15T00:57:41.167 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:41.168 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:41.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff2fc065430 msgr2=0x7ff2fc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:41.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff2fc065430 0x7ff2fc0678e0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 crypto rx=0x7ff30c02a600 tx=0x7ff30c035040 comp rx=0 tx=0).stop 2024-06-15T00:57:41.169 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 msgr2=0x7ff310102590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 0x7ff310102590 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 crypto rx=0x7ff30000b790 tx=0x7ff3000091d0 comp rx=0 tx=0).stop 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 shutdown_connections 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7ff2fc065430 0x7ff2fc0678e0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7ff310101b70 0x7ff310102590 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 --2- 172.21.15.105:0/2169303028 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7ff3101011d0 0x7ff310102050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 >> 172.21.15.105:0/2169303028 conn(0x7ff3100f6d80 msgr2=0x7ff3100f7d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 shutdown_connections 2024-06-15T00:57:41.170 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:41.169+0000 7ff317a8d640 1 -- 172.21.15.105:0/2169303028 wait complete. 2024-06-15T00:57:41.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:41 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2169303028' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:41.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:41 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2169303028' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:42.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:42 smithi105 ceph-mon[28877]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:42.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:42 smithi143 ceph-mon[35860]: pgmap v277: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:44.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:44 smithi105 ceph-mon[28877]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:44.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:44 smithi143 ceph-mon[35860]: pgmap v278: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:44.730 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:44.995 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:46.362 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:46 smithi105 ceph-mon[28877]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:46.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 -- 172.21.15.105:0/1998079382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fd320 msgr2=0x7f76f40fd6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:46.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 --2- 172.21.15.105:0/1998079382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fd320 0x7f76f40fd6f0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 crypto rx=0x7f76e40048d0 tx=0x7f76e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 -- 172.21.15.105:0/1998079382 shutdown_connections 2024-06-15T00:57:46.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 --2- 172.21.15.105:0/1998079382 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fdcc0 0x7f76f4106390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 --2- 172.21.15.105:0/1998079382 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fd320 0x7f76f40fd6f0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.565 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 -- 172.21.15.105:0/1998079382 >> 172.21.15.105:0/1998079382 conn(0x7f76f40f6dc0 msgr2=0x7f76f40f91d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 -- 172.21.15.105:0/1998079382 shutdown_connections 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.557+0000 7f76f9c72640 1 -- 172.21.15.105:0/1998079382 wait complete. 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f9c72640 1 Processor -- start 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f9c72640 1 -- start start 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f9c72640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fd320 0x7f76f40fb1e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f9c72640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 0x7f76f40fb720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f9c72640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f40fbcf0 con 0x7f76f40fdcc0 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f9c72640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f40fbe60 con 0x7f76f40fd320 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 0x7f76f40fb720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f2ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 0x7f76f40fb720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40512/0 (socket says 172.21.15.105:40512) 2024-06-15T00:57:46.566 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f2ffd640 1 -- 172.21.15.105:0/3498011134 learned_addr learned my addr 172.21.15.105:0/3498011134 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.558+0000 7f76f37fe640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fd320 0x7f76f40fb1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f2ffd640 1 -- 172.21.15.105:0/3498011134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fd320 msgr2=0x7f76f40fb1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f2ffd640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fd320 0x7f76f40fb1e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f2ffd640 1 -- 172.21.15.105:0/3498011134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e4005040 con 0x7f76f40fdcc0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f37fe640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fd320 0x7f76f40fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_reply_more state changed! 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f2ffd640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 0x7f76f40fb720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f76e800b790 tx=0x7f76e80091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e800c700 con 0x7f76f40fdcc0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f4119340 con 0x7f76f40fdcc0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76e800cd10 con 0x7f76f40fdcc0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f76e8013370 con 0x7f76f40fdcc0 2024-06-15T00:57:46.567 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.559+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f4119780 con 0x7f76f40fdcc0 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.560+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f4065ff0 con 0x7f76f40fdcc0 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.560+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f76e8012070 con 0x7f76f40fdcc0 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.561+0000 7f76dbfff640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f76dc065430 0x7f76dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.561+0000 7f76f37fe640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f76dc065430 0x7f76dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.561+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f76e8082170 con 0x7f76f40fdcc0 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.561+0000 7f76f37fe640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f76dc065430 0x7f76dc0678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f76e4007ac0 tx=0x7f76e4035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:46.568 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.565+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f76e8050620 con 0x7f76f40fdcc0 2024-06-15T00:57:46.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:46 smithi143 ceph-mon[35860]: pgmap v279: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:46.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.761+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f76f40fa9c0 con 0x7f76f40fdcc0 2024-06-15T00:57:46.762 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.762+0000 7f76dbfff640 1 -- 172.21.15.105:0/3498011134 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f76e8053c40 con 0x7f76f40fdcc0 2024-06-15T00:57:46.762 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:46.765 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:46.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f76dc065430 msgr2=0x7f76dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:46.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f76dc065430 0x7f76dc0678e0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 crypto rx=0x7f76e4007ac0 tx=0x7f76e4035040 comp rx=0 tx=0).stop 2024-06-15T00:57:46.766 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 msgr2=0x7f76f40fb720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 0x7f76f40fb720 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 crypto rx=0x7f76e800b790 tx=0x7f76e80091d0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 shutdown_connections 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f76dc065430 0x7f76dc0678e0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f76f40fdcc0 0x7f76f40fb720 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 --2- 172.21.15.105:0/3498011134 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f76f40fd320 0x7f76f40fb1e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 >> 172.21.15.105:0/3498011134 conn(0x7f76f40f6dc0 msgr2=0x7f76f40f8330 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 shutdown_connections 2024-06-15T00:57:46.767 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:46.764+0000 7f76f9c72640 1 -- 172.21.15.105:0/3498011134 wait complete. 2024-06-15T00:57:47.408 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:47 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/3498011134' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:47.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:47 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/3498011134' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:48.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:48 smithi105 ceph-mon[28877]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:48.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:48 smithi143 ceph-mon[35860]: pgmap v280: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:50.410 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:50.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:50 smithi105 ceph-mon[28877]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:50.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:50 smithi143 ceph-mon[35860]: pgmap v281: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:50.677 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:52.344 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:52 smithi105 ceph-mon[28877]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:52.344 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:52 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:57:52.389 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 -- 172.21.15.105:0/2926969109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fbe80 msgr2=0x7f02fc0fcd80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 --2- 172.21.15.105:0/2926969109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fbe80 0x7f02fc0fcd80 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 crypto rx=0x7f02e40048d0 tx=0x7f02e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 -- 172.21.15.105:0/2926969109 shutdown_connections 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 --2- 172.21.15.105:0/2926969109 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fbe80 0x7f02fc0fcd80 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 --2- 172.21.15.105:0/2926969109 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fb570 0x7f02fc0fb940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 -- 172.21.15.105:0/2926969109 >> 172.21.15.105:0/2926969109 conn(0x7f02fc072110 msgr2=0x7f02fc072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 -- 172.21.15.105:0/2926969109 shutdown_connections 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.343+0000 7f030191e640 1 -- 172.21.15.105:0/2926969109 wait complete. 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f030191e640 1 Processor -- start 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f030191e640 1 -- start start 2024-06-15T00:57:52.390 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f030191e640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 0x7f02fc10dec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f030191e640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fbe80 0x7f02fc10e400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f030191e640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc10ea50 con 0x7f02fc0fb570 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f030191e640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02fc10ebc0 con 0x7f02fc0fbe80 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02faffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 0x7f02fc10dec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02faffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 0x7f02fc10dec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40534/0 (socket says 172.21.15.105:40534) 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02faffd640 1 -- 172.21.15.105:0/749892252 learned_addr learned my addr 172.21.15.105:0/749892252 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02f3fff640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fbe80 0x7f02fc10e400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02faffd640 1 -- 172.21.15.105:0/749892252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fbe80 msgr2=0x7f02fc10e400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02faffd640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fbe80 0x7f02fc10e400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02faffd640 1 -- 172.21.15.105:0/749892252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02e4005040 con 0x7f02fc0fb570 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.344+0000 7f02f3fff640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fbe80 0x7f02fc10e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_auth_done state changed! 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.345+0000 7f02faffd640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 0x7f02fc10dec0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00eb70 tx=0x7f02ec00b040 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:52.391 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.345+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02ec007c30 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.345+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02fc116f30 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.345+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02ec00f040 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.345+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f02ec009470 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.345+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02fc1173f0 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.346+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02fc0fc920 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.347+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f02ec0096e0 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.347+0000 7f02f37fe640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f02e8065430 0x7f02e80678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.347+0000 7f02f3fff640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f02e8065430 0x7f02e80678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.347+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f02ec082100 con 0x7f02fc0fb570 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.348+0000 7f02f3fff640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f02e8065430 0x7f02e80678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f02e4003080 tx=0x7f02e4035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:52.392 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.351+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f02ec050660 con 0x7f02fc0fb570 2024-06-15T00:57:52.561 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.560+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f02fc072c20 con 0x7f02fc0fb570 2024-06-15T00:57:52.561 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.561+0000 7f02f37fe640 1 -- 172.21.15.105:0/749892252 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f02ec053c80 con 0x7f02fc0fb570 2024-06-15T00:57:52.561 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:52.561 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f02e8065430 msgr2=0x7f02e80678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f02e8065430 0x7f02e80678e0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 crypto rx=0x7f02e4003080 tx=0x7f02e4035040 comp rx=0 tx=0).stop 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 msgr2=0x7f02fc10dec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 0x7f02fc10dec0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 crypto rx=0x7f02ec00eb70 tx=0x7f02ec00b040 comp rx=0 tx=0).stop 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 shutdown_connections 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f02e8065430 0x7f02e80678e0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.563 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f02fc0fbe80 0x7f02fc10e400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 --2- 172.21.15.105:0/749892252 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f02fc0fb570 0x7f02fc10dec0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:52.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 >> 172.21.15.105:0/749892252 conn(0x7f02fc072110 msgr2=0x7f02fc0ffea0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:52.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 shutdown_connections 2024-06-15T00:57:52.564 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:52.563+0000 7f030191e640 1 -- 172.21.15.105:0/749892252 wait complete. 2024-06-15T00:57:52.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:52 smithi143 ceph-mon[35860]: pgmap v282: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:52.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:52 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-06-15T00:57:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:53 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/749892252' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:57:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:57:53.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:53 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:57:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:53 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/749892252' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-06-15T00:57:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "auth get", "entity": "client.admin"}]: dispatch 2024-06-15T00:57:53.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:53 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' 2024-06-15T00:57:54.482 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:54 smithi105 ceph-mon[28877]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:54.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:54 smithi143 ceph-mon[35860]: pgmap v283: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:55.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:55 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:55.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:55 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:57:56.126 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:57:56.391 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:57:56.510 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:56 smithi105 ceph-mon[28877]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:56.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:56 smithi143 ceph-mon[35860]: pgmap v284: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:58.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 -- 172.21.15.105:0/488390764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 msgr2=0x7f66f4102670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:58.015 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 --2- 172.21.15.105:0/488390764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4102670 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 crypto rx=0x7f66e40048d0 tx=0x7f66e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 -- 172.21.15.105:0/488390764 shutdown_connections 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 --2- 172.21.15.105:0/488390764 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f4102c40 0x7f66f40fc2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 --2- 172.21.15.105:0/488390764 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4102670 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 -- 172.21.15.105:0/488390764 >> 172.21.15.105:0/488390764 conn(0x7f66f40f7e50 msgr2=0x7f66f40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 -- 172.21.15.105:0/488390764 shutdown_connections 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.012+0000 7f66f8993640 1 -- 172.21.15.105:0/488390764 wait complete. 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f8993640 1 Processor -- start 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f8993640 1 -- start start 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f8993640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f8993640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f4102c40 0x7f66f4113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f8993640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f4113b90 con 0x7f66f41022a0 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f8993640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66f410cf40 con 0x7f66f4102c40 2024-06-15T00:57:58.016 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f259c640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4112fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:33534/0 (socket says 172.21.15.105:33534) 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f259c640 1 -- 172.21.15.105:0/1742192982 learned_addr learned my addr 172.21.15.105:0/1742192982 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f259c640 1 -- 172.21.15.105:0/1742192982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f4102c40 msgr2=0x7f66f4113500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f259c640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f4102c40 0x7f66f4113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.013+0000 7f66f259c640 1 -- 172.21.15.105:0/1742192982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66e4005040 con 0x7f66f41022a0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.014+0000 7f66f259c640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4112fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f66e402a600 tx=0x7f66e4009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.014+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66e4038070 con 0x7f66f41022a0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.014+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66e40096f0 con 0x7f66f41022a0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.014+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66f410d1c0 con 0x7f66f41022a0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.014+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f66e4032cd0 con 0x7f66f41022a0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.014+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66f410d620 con 0x7f66f41022a0 2024-06-15T00:57:58.017 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.015+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66f40fda10 con 0x7f66f41022a0 2024-06-15T00:57:58.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.015+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f66e4030020 con 0x7f66f41022a0 2024-06-15T00:57:58.018 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.016+0000 7f66eaffd640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f66e0065480 0x7f66e0067930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:57:58.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.016+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f66e40a24c0 con 0x7f66f41022a0 2024-06-15T00:57:58.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.016+0000 7f66ebfff640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f66e0065480 0x7f66e0067930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:57:58.019 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.016+0000 7f66ebfff640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f66e0065480 0x7f66e0067930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f66f410e880 tx=0x7f66dc006d20 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:57:58.020 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.020+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f66e4070970 con 0x7f66f41022a0 2024-06-15T00:57:58.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.218+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f66f40fba90 con 0x7f66f41022a0 2024-06-15T00:57:58.219 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.218+0000 7f66eaffd640 1 -- 172.21.15.105:0/1742192982 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f66e4073f90 con 0x7f66f41022a0 2024-06-15T00:57:58.219 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:57:58.219 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:57:58.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f66e0065480 msgr2=0x7f66e0067930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:58.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f66e0065480 0x7f66e0067930 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 crypto rx=0x7f66f410e880 tx=0x7f66dc006d20 comp rx=0 tx=0).stop 2024-06-15T00:57:58.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 msgr2=0x7f66f4112fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:57:58.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4112fc0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 crypto rx=0x7f66e402a600 tx=0x7f66e4009da0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.222 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 shutdown_connections 2024-06-15T00:57:58.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f66e0065480 0x7f66e0067930 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66f4102c40 0x7f66f4113500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 --2- 172.21.15.105:0/1742192982 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66f41022a0 0x7f66f4112fc0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:57:58.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 >> 172.21.15.105:0/1742192982 conn(0x7f66f40f7e50 msgr2=0x7f66f40f8dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:57:58.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 shutdown_connections 2024-06-15T00:57:58.223 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:57:58.221+0000 7f66f8993640 1 -- 172.21.15.105:0/1742192982 wait complete. 2024-06-15T00:57:58.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:58 smithi105 ceph-mon[28877]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:58.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:57:58 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1742192982' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:57:58.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:58 smithi143 ceph-mon[35860]: pgmap v285: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:57:58.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:57:58 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1742192982' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:00.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:00 smithi105 ceph-mon[28877]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:00.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:00 smithi143 ceph-mon[35860]: pgmap v286: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:01.958 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:02.221 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:02.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:02 smithi105 ceph-mon[28877]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:02.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:02 smithi143 ceph-mon[35860]: pgmap v287: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 -- 172.21.15.105:0/2670635835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7981011a0 msgr2=0x7fc798101570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 --2- 172.21.15.105:0/2670635835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7981011a0 0x7fc798101570 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 crypto rx=0x7fc788004810 tx=0x7fc788009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 -- 172.21.15.105:0/2670635835 shutdown_connections 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 --2- 172.21.15.105:0/2670635835 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc798101b40 0x7fc7980fb1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 --2- 172.21.15.105:0/2670635835 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc7981011a0 0x7fc798101570 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 -- 172.21.15.105:0/2670635835 >> 172.21.15.105:0/2670635835 conn(0x7fc7980f6d90 msgr2=0x7fc7980f91a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:03.935 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 -- 172.21.15.105:0/2670635835 shutdown_connections 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 -- 172.21.15.105:0/2670635835 wait complete. 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 Processor -- start 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.910+0000 7fc79ce17640 1 -- start start 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc79ce17640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7981011a0 0x7fc798111de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc79ce17640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 0x7fc798112320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc79ce17640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc798112920 con 0x7fc798101b40 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc79ce17640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc79810bd60 con 0x7fc7981011a0 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 0x7fc798112320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 0x7fc798112320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52130/0 (socket says 172.21.15.105:52130) 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 -- 172.21.15.105:0/1892700419 learned_addr learned my addr 172.21.15.105:0/1892700419 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:03.936 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc79659c640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7981011a0 0x7fc798111de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 -- 172.21.15.105:0/1892700419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7981011a0 msgr2=0x7fc798111de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7981011a0 0x7fc798111de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 -- 172.21.15.105:0/1892700419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc780005040 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78ffff640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 0x7fc798112320 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc7800094d0 tx=0x7fc780009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc78000e8b0 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc78000f040 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.911+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc788005040 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.912+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7fc780013da0 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.912+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc79810c450 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.913+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc7980fc910 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.913+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7fc780013590 con 0x7fc798101b40 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.914+0000 7fc78effd640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc784065430 0x7fc7840678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:03.937 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.914+0000 7fc79659c640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc784065430 0x7fc7840678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:03.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.914+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7fc7800821e0 con 0x7fc798101b40 2024-06-15T00:58:03.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.914+0000 7fc79659c640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc784065430 0x7fc7840678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc788000c00 tx=0x7fc788035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:03.938 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:03.918+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7fc780050710 con 0x7fc798101b40 2024-06-15T00:58:04.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.121+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fc7980fa990 con 0x7fc798101b40 2024-06-15T00:58:04.122 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.121+0000 7fc78effd640 1 -- 172.21.15.105:0/1892700419 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7fc780053d30 con 0x7fc798101b40 2024-06-15T00:58:04.122 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:04.122 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc784065430 msgr2=0x7fc7840678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc784065430 0x7fc7840678e0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 crypto rx=0x7fc788000c00 tx=0x7fc788035040 comp rx=0 tx=0).stop 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 msgr2=0x7fc798112320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 0x7fc798112320 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 crypto rx=0x7fc7800094d0 tx=0x7fc780009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 shutdown_connections 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7fc784065430 0x7fc7840678e0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7fc798101b40 0x7fc798112320 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 --2- 172.21.15.105:0/1892700419 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7fc7981011a0 0x7fc798111de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 >> 172.21.15.105:0/1892700419 conn(0x7fc7980f6d90 msgr2=0x7fc7980f7d10 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 shutdown_connections 2024-06-15T00:58:04.124 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:04.123+0000 7fc79ce17640 1 -- 172.21.15.105:0/1892700419 wait complete. 2024-06-15T00:58:04.461 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:04 smithi105 ceph-mon[28877]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:04.461 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:04 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1892700419' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:04.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:04 smithi143 ceph-mon[35860]: pgmap v288: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:04.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:04 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1892700419' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:06.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:06 smithi105 ceph-mon[28877]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:06.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:06 smithi143 ceph-mon[35860]: pgmap v289: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:07.839 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:08.104 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:08.526 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:08 smithi105 ceph-mon[28877]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:08.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:08 smithi143 ceph-mon[35860]: pgmap v290: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 -- 172.21.15.105:0/3846922794 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 msgr2=0x7f53f00fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 --2- 172.21.15.105:0/3846922794 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f00fb1e0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 crypto rx=0x7f53e00048d0 tx=0x7f53e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 -- 172.21.15.105:0/3846922794 shutdown_connections 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 --2- 172.21.15.105:0/3846922794 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f00fb1e0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 --2- 172.21.15.105:0/3846922794 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53f01011b0 0x7f53f0101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 -- 172.21.15.105:0/3846922794 >> 172.21.15.105:0/3846922794 conn(0x7f53f00f6d60 msgr2=0x7f53f00f9170 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 -- 172.21.15.105:0/3846922794 shutdown_connections 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.595+0000 7f53f7e92640 1 -- 172.21.15.105:0/3846922794 wait complete. 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f7e92640 1 Processor -- start 2024-06-15T00:58:09.597 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f7e92640 1 -- start start 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f7e92640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53f01011b0 0x7f53f0111ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f7e92640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f0112410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f7e92640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0112aa0 con 0x7f53f0101b50 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f7e92640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f010be50 con 0x7f53f01011b0 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f542d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f0112410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f542d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f0112410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:52152/0 (socket says 172.21.15.105:52152) 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f542d640 1 -- 172.21.15.105:0/1086143092 learned_addr learned my addr 172.21.15.105:0/1086143092 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f5c2e640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53f01011b0 0x7f53f0111ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f542d640 1 -- 172.21.15.105:0/1086143092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53f01011b0 msgr2=0x7f53f0111ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f542d640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53f01011b0 0x7f53f0111ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.596+0000 7f53f542d640 1 -- 172.21.15.105:0/1086143092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0005040 con 0x7f53f0101b50 2024-06-15T00:58:09.598 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.597+0000 7f53f542d640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f0112410 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f53e00048a0 tx=0x7f53e0009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:09.599 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.597+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e0038070 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.597+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f010c0d0 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.597+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e0032d30 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.597+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f53e0009680 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.597+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f010c590 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.598+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f00fc920 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.598+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f53e0030020 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.599+0000 7f53e67fc640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f53dc065430 0x7f53dc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.599+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f53e00a2250 con 0x7f53f0101b50 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.599+0000 7f53f5c2e640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f53dc065430 0x7f53dc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:09.600 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.599+0000 7f53f5c2e640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f53dc065430 0x7f53dc0678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009b50 tx=0x7f53ec009120 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:09.603 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.603+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f53e00707b0 con 0x7f53f0101b50 2024-06-15T00:58:09.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.804+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f53f0108260 con 0x7f53f0101b50 2024-06-15T00:58:09.805 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.805+0000 7f53e67fc640 1 -- 172.21.15.105:0/1086143092 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f53e0073dd0 con 0x7f53f0101b50 2024-06-15T00:58:09.805 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:09.805 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:09.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f53dc065430 msgr2=0x7f53dc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:09.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f53dc065430 0x7f53dc0678e0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 crypto rx=0x7f53ec009b50 tx=0x7f53ec009120 comp rx=0 tx=0).stop 2024-06-15T00:58:09.807 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 msgr2=0x7f53f0112410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f0112410 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 crypto rx=0x7f53e00048a0 tx=0x7f53e0009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 shutdown_connections 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f53dc065430 0x7f53dc0678e0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f53f0101b50 0x7f53f0112410 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 --2- 172.21.15.105:0/1086143092 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f53f01011b0 0x7f53f0111ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 >> 172.21.15.105:0/1086143092 conn(0x7f53f00f6d60 msgr2=0x7f53f00f7d00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:09.808 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 shutdown_connections 2024-06-15T00:58:09.809 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:09.807+0000 7f53f7e92640 1 -- 172.21.15.105:0/1086143092 wait complete. 2024-06-15T00:58:10.500 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:10 smithi105 ceph-mon[28877]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:10.500 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:10 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:58:10.500 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:10 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/1086143092' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:10.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:10 smithi143 ceph-mon[35860]: pgmap v291: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:10.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:10 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:58:10.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:10 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/1086143092' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:12.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:12 smithi105 ceph-mon[28877]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:12.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:12 smithi143 ceph-mon[35860]: pgmap v292: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:13.502 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:13.766 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:14.481 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:14 smithi105 ceph-mon[28877]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:14.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:14 smithi143 ceph-mon[35860]: pgmap v293: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 -- 172.21.15.105:0/1907808372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 msgr2=0x7f19f40fc2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 --2- 172.21.15.105:0/1907808372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f40fc2d0 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0048d0 tx=0x7f19dc009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 -- 172.21.15.105:0/1907808372 shutdown_connections 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 --2- 172.21.15.105:0/1907808372 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f40fc2d0 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 --2- 172.21.15.105:0/1907808372 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 0x7f19f4102670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 -- 172.21.15.105:0/1907808372 >> 172.21.15.105:0/1907808372 conn(0x7f19f40f7e50 msgr2=0x7f19f40fa260 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 -- 172.21.15.105:0/1907808372 shutdown_connections 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 -- 172.21.15.105:0/1907808372 wait complete. 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 Processor -- start 2024-06-15T00:58:15.428 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.427+0000 7f19f9b04640 1 -- start start 2024-06-15T00:58:15.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19f9b04640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 0x7f19f4112f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:15.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19f9b04640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f41134c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:15.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19f9b04640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f4113b50 con 0x7f19f4102c40 2024-06-15T00:58:15.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19f9b04640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f410cf00 con 0x7f19f41022a0 2024-06-15T00:58:15.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f41134c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:15.430 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f41134c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50520/0 (socket says 172.21.15.105:50520) 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 -- 172.21.15.105:0/215244798 learned_addr learned my addr 172.21.15.105:0/215244798 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19f37fe640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 0x7f19f4112f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 -- 172.21.15.105:0/215244798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 msgr2=0x7f19f4112f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 0x7f19f4112f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 -- 172.21.15.105:0/215244798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19dc005040 con 0x7f19f4102c40 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19f37fe640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 0x7f19f4112f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.428+0000 7f19eaffd640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f41134c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0048a0 tx=0x7f19dc009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.429+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19dc038070 con 0x7f19f4102c40 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.429+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19dc032d30 con 0x7f19f4102c40 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.429+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f410d180 con 0x7f19f4102c40 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.429+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f19dc009680 con 0x7f19f4102c40 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.429+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f410d640 con 0x7f19f4102c40 2024-06-15T00:58:15.431 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.430+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f40fda10 con 0x7f19f4102c40 2024-06-15T00:58:15.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.430+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f19dc030020 con 0x7f19f4102c40 2024-06-15T00:58:15.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.430+0000 7f19f0ff9640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f19e0065430 0x7f19e00678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:15.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.431+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f19dc0a2300 con 0x7f19f4102c40 2024-06-15T00:58:15.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.431+0000 7f19f37fe640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f19e0065430 0x7f19e00678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:15.432 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.431+0000 7f19f37fe640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f19e0065430 0x7f19e00678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f19e4006fd0 tx=0x7f19e4008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:15.435 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.435+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f19dc0707b0 con 0x7f19f4102c40 2024-06-15T00:58:15.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.632+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f19f4109380 con 0x7f19f4102c40 2024-06-15T00:58:15.633 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.632+0000 7f19f0ff9640 1 -- 172.21.15.105:0/215244798 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f19dc073dd0 con 0x7f19f4102c40 2024-06-15T00:58:15.633 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:15.633 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f19e0065430 msgr2=0x7f19e00678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f19e0065430 0x7f19e00678e0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 crypto rx=0x7f19e4006fd0 tx=0x7f19e4008040 comp rx=0 tx=0).stop 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 msgr2=0x7f19f41134c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f41134c0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 crypto rx=0x7f19dc0048a0 tx=0x7f19dc009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 shutdown_connections 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f19e0065430 0x7f19e00678e0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f19f4102c40 0x7f19f41134c0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 --2- 172.21.15.105:0/215244798 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f19f41022a0 0x7f19f4112f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.634+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 >> 172.21.15.105:0/215244798 conn(0x7f19f40f7e50 msgr2=0x7f19f40f8e20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:15.635 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.635+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 shutdown_connections 2024-06-15T00:58:15.636 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:15.635+0000 7f19f9b04640 1 -- 172.21.15.105:0/215244798 wait complete. 2024-06-15T00:58:16.339 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:16 smithi105 ceph-mon[28877]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:16.339 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:16 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/215244798' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:16.638 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:16 smithi143 ceph-mon[35860]: pgmap v294: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:16.638 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:16 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/215244798' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:18.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:18 smithi105 ceph-mon[28877]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:18.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:18 smithi143 ceph-mon[35860]: pgmap v295: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:19.366 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:19.627 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:20.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:20 smithi105 ceph-mon[28877]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:20.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:20 smithi143 ceph-mon[35860]: pgmap v296: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:21.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 -- 172.21.15.105:0/901533594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 msgr2=0x7f857c0fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:21.178 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 --2- 172.21.15.105:0/901533594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c0fb1e0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 crypto rx=0x7f856c009c50 tx=0x7f856c004d10 comp rx=0 tx=0).stop 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 -- 172.21.15.105:0/901533594 shutdown_connections 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 --2- 172.21.15.105:0/901533594 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c0fb1e0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 --2- 172.21.15.105:0/901533594 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 0x7f857c101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 -- 172.21.15.105:0/901533594 >> 172.21.15.105:0/901533594 conn(0x7f857c0f6d80 msgr2=0x7f857c0f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 -- 172.21.15.105:0/901533594 shutdown_connections 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 -- 172.21.15.105:0/901533594 wait complete. 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 Processor -- start 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.176+0000 7f8582c4f640 1 -- start start 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f8582c4f640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 0x7f857c112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f8582c4f640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c1134f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f8582c4f640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c113b80 con 0x7f857c101b50 2024-06-15T00:58:21.179 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f8582c4f640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f857c10cf30 con 0x7f857c1011b0 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c1134f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c1134f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:50532/0 (socket says 172.21.15.105:50532) 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 -- 172.21.15.105:0/2117396681 learned_addr learned my addr 172.21.15.105:0/2117396681 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f85809eb640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 0x7f857c112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 -- 172.21.15.105:0/2117396681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 msgr2=0x7f857c112fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 0x7f857c112fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 -- 172.21.15.105:0/2117396681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f856c0098b0 con 0x7f857c101b50 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f85809eb640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 0x7f857c112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.177+0000 7f857bfff640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c1134f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f856c003400 tx=0x7f856c004d10 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.178+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f856c041070 con 0x7f857c101b50 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.178+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f857c10d1b0 con 0x7f857c101b50 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.178+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f856c009690 con 0x7f857c101b50 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.178+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f856c03d8c0 con 0x7f857c101b50 2024-06-15T00:58:21.180 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.178+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f857c10d670 con 0x7f857c101b50 2024-06-15T00:58:21.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.179+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f857c0fc920 con 0x7f857c101b50 2024-06-15T00:58:21.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.179+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f856c02f030 con 0x7f857c101b50 2024-06-15T00:58:21.181 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.179+0000 7f85797fa640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8568065430 0x7f85680678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:21.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.180+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f856c0a3140 con 0x7f857c101b50 2024-06-15T00:58:21.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.180+0000 7f85809eb640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8568065430 0x7f85680678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:21.182 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.180+0000 7f85809eb640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8568065430 0x7f85680678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8574006fd0 tx=0x7f8574008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:21.184 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.184+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f856c0715f0 con 0x7f857c101b50 2024-06-15T00:58:21.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.383+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f857c0fa9a0 con 0x7f857c101b50 2024-06-15T00:58:21.384 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.383+0000 7f85797fa640 1 -- 172.21.15.105:0/2117396681 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f856c074c10 con 0x7f857c101b50 2024-06-15T00:58:21.384 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:21.384 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8568065430 msgr2=0x7f85680678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8568065430 0x7f85680678e0 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 crypto rx=0x7f8574006fd0 tx=0x7f8574008040 comp rx=0 tx=0).stop 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 msgr2=0x7f857c1134f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c1134f0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 crypto rx=0x7f856c003400 tx=0x7f856c004d10 comp rx=0 tx=0).stop 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 shutdown_connections 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f8568065430 0x7f85680678e0 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f857c101b50 0x7f857c1134f0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 --2- 172.21.15.105:0/2117396681 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f857c1011b0 0x7f857c112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 >> 172.21.15.105:0/2117396681 conn(0x7f857c0f6d80 msgr2=0x7f857c0f7dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 shutdown_connections 2024-06-15T00:58:21.386 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:21.385+0000 7f8582c4f640 1 -- 172.21.15.105:0/2117396681 wait complete. 2024-06-15T00:58:22.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:22 smithi105 ceph-mon[28877]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:22.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:22 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2117396681' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:22.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:22 smithi143 ceph-mon[35860]: pgmap v297: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:22.646 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:22 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2117396681' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:24.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:24 smithi105 ceph-mon[28877]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:24.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:24 smithi143 ceph-mon[35860]: pgmap v298: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:24.903 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:25.166 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:25.523 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:25 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:58:25.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:25 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:58:26.606 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:26 smithi105 ceph-mon[28877]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:26.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:26 smithi143 ceph-mon[35860]: pgmap v299: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 -- 172.21.15.105:0/3874994477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fb570 msgr2=0x7f66540fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 --2- 172.21.15.105:0/3874994477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fb570 0x7f66540fb940 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 crypto rx=0x7f6644009c50 tx=0x7f6644004d10 comp rx=0 tx=0).stop 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 -- 172.21.15.105:0/3874994477 shutdown_connections 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 --2- 172.21.15.105:0/3874994477 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66540fbe80 0x7f66540fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 --2- 172.21.15.105:0/3874994477 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fb570 0x7f66540fb940 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 -- 172.21.15.105:0/3874994477 >> 172.21.15.105:0/3874994477 conn(0x7f6654072110 msgr2=0x7f6654072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 -- 172.21.15.105:0/3874994477 shutdown_connections 2024-06-15T00:58:26.793 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 -- 172.21.15.105:0/3874994477 wait complete. 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 Processor -- start 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.791+0000 7f6659d9d640 1 -- start start 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6659d9d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66540fb570 0x7f665410df60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6659d9d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 0x7f665410e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6659d9d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665410eb80 con 0x7f66540fbe80 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6659d9d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f665410ecf0 con 0x7f66540fb570 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 0x7f665410e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 0x7f665410e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53778/0 (socket says 172.21.15.105:53778) 2024-06-15T00:58:26.794 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 -- 172.21.15.105:0/17871408 learned_addr learned my addr 172.21.15.105:0/17871408 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f66537fe640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66540fb570 0x7f665410df60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 -- 172.21.15.105:0/17871408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66540fb570 msgr2=0x7f665410df60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66540fb570 0x7f665410df60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 -- 172.21.15.105:0/17871408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66440098b0 con 0x7f66540fbe80 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6652ffd640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 0x7f665410e4a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f663c00b790 tx=0x7f663c0091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663c00c700 con 0x7f66540fbe80 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f663c00cd10 con 0x7f66540fbe80 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6654116fd0 con 0x7f66540fbe80 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f663c013370 con 0x7f66540fbe80 2024-06-15T00:58:26.795 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.792+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66541174c0 con 0x7f66540fbe80 2024-06-15T00:58:26.797 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.793+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66540fc920 con 0x7f66540fbe80 2024-06-15T00:58:26.797 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.794+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f663c012070 con 0x7f66540fbe80 2024-06-15T00:58:26.797 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.794+0000 7f664bfff640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6640065430 0x7f66400678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:26.797 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.794+0000 7f66537fe640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6640065430 0x7f66400678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:26.797 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.794+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f663c082150 con 0x7f66540fbe80 2024-06-15T00:58:26.797 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.795+0000 7f66537fe640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6640065430 0x7f66400678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6644009c20 tx=0x7f6644004d10 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:26.799 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.798+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f663c050600 con 0x7f66540fbe80 2024-06-15T00:58:26.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.999+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f6654072c20 con 0x7f66540fbe80 2024-06-15T00:58:26.999 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:26.999+0000 7f664bfff640 1 -- 172.21.15.105:0/17871408 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f663c053c20 con 0x7f66540fbe80 2024-06-15T00:58:27.000 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:27.000 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6640065430 msgr2=0x7f66400678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6640065430 0x7f66400678e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 crypto rx=0x7f6644009c20 tx=0x7f6644004d10 comp rx=0 tx=0).stop 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 msgr2=0x7f665410e4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 0x7f665410e4a0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 crypto rx=0x7f663c00b790 tx=0x7f663c0091d0 comp rx=0 tx=0).stop 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 shutdown_connections 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f6640065430 0x7f66400678e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f66540fbe80 0x7f665410e4a0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:27.002 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 --2- 172.21.15.105:0/17871408 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f66540fb570 0x7f665410df60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:27.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.001+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 >> 172.21.15.105:0/17871408 conn(0x7f6654072110 msgr2=0x7f66540f8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:27.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.002+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 shutdown_connections 2024-06-15T00:58:27.003 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:27.002+0000 7f6659d9d640 1 -- 172.21.15.105:0/17871408 wait complete. 2024-06-15T00:58:27.543 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:27 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/17871408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:27.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:27 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/17871408' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:28.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:28 smithi143 ceph-mon[35860]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:28.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:28 smithi105 ceph-mon[28877]: pgmap v300: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:30.568 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:30.597 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:30 smithi105 ceph-mon[28877]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:30.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:30 smithi143 ceph-mon[35860]: pgmap v301: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:30.833 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:32.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.445+0000 7efc763ed640 1 -- 172.21.15.105:0/383155701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 msgr2=0x7efc700fb940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:32.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.445+0000 7efc763ed640 1 --2- 172.21.15.105:0/383155701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc700fb940 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 crypto rx=0x7efc600048d0 tx=0x7efc60009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.448 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.445+0000 7efc763ed640 1 -- 172.21.15.105:0/383155701 shutdown_connections 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.445+0000 7efc763ed640 1 --2- 172.21.15.105:0/383155701 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc700fbe80 0x7efc700fcd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.445+0000 7efc763ed640 1 --2- 172.21.15.105:0/383155701 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc700fb940 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.445+0000 7efc763ed640 1 -- 172.21.15.105:0/383155701 >> 172.21.15.105:0/383155701 conn(0x7efc70072110 msgr2=0x7efc70072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 -- 172.21.15.105:0/383155701 shutdown_connections 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 -- 172.21.15.105:0/383155701 wait complete. 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 Processor -- start 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 -- start start 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc70112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc700fbe80 0x7efc701134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc70113b70 con 0x7efc700fb570 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.446+0000 7efc763ed640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc7010cf20 con 0x7efc700fbe80 2024-06-15T00:58:32.449 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc70112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc70112fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:53798/0 (socket says 172.21.15.105:53798) 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 -- 172.21.15.105:0/797439891 learned_addr learned my addr 172.21.15.105:0/797439891 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6f7fe640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc700fbe80 0x7efc701134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 -- 172.21.15.105:0/797439891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc700fbe80 msgr2=0x7efc701134e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc700fbe80 0x7efc701134e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 -- 172.21.15.105:0/797439891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc60005040 con 0x7efc700fb570 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6ffff640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc70112fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc60009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc60038070 con 0x7efc700fb570 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc7010d1a0 con 0x7efc700fb570 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc600096f0 con 0x7efc700fb570 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc60032da0 con 0x7efc700fb570 2024-06-15T00:58:32.450 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.447+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc7010d660 con 0x7efc700fb570 2024-06-15T00:58:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.448+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc700fc920 con 0x7efc700fb570 2024-06-15T00:58:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.449+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7efc60030020 con 0x7efc700fb570 2024-06-15T00:58:32.451 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.450+0000 7efc6cff9640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc58065430 0x7efc580678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.450+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc600a2990 con 0x7efc700fb570 2024-06-15T00:58:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.450+0000 7efc6f7fe640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc58065430 0x7efc580678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:32.452 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.451+0000 7efc6f7fe640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc58065430 0x7efc580678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc7010e860 tx=0x7efc64009120 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:32.454 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.454+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efc60070ec0 con 0x7efc700fb570 2024-06-15T00:58:32.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:32 smithi105 ceph-mon[28877]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:32.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:32 smithi143 ceph-mon[35860]: pgmap v302: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:32.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.652+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc70072c20 con 0x7efc700fb570 2024-06-15T00:58:32.653 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.653+0000 7efc6cff9640 1 -- 172.21.15.105:0/797439891 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7efc600744e0 con 0x7efc700fb570 2024-06-15T00:58:32.653 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:32.654 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:32.655 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc58065430 msgr2=0x7efc580678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:32.655 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc58065430 0x7efc580678e0 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 crypto rx=0x7efc7010e860 tx=0x7efc64009120 comp rx=0 tx=0).stop 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 msgr2=0x7efc70112fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc70112fa0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 crypto rx=0x7efc60000c00 tx=0x7efc60009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 shutdown_connections 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc58065430 0x7efc580678e0 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc700fbe80 0x7efc701134e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 --2- 172.21.15.105:0/797439891 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc700fb570 0x7efc70112fa0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 >> 172.21.15.105:0/797439891 conn(0x7efc70072110 msgr2=0x7efc700f8720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 shutdown_connections 2024-06-15T00:58:32.656 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:32.655+0000 7efc763ed640 1 -- 172.21.15.105:0/797439891 wait complete. 2024-06-15T00:58:33.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:33 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/797439891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:33.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:33 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/797439891' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:34.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:34 smithi143 ceph-mon[35860]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:34.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:34 smithi105 ceph-mon[28877]: pgmap v303: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:36.344 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:36.605 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:36.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:36 smithi143 ceph-mon[35860]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:36.728 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:36 smithi105 ceph-mon[28877]: pgmap v304: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 -- 172.21.15.105:0/3754210381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 msgr2=0x7f7030101590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 --2- 172.21.15.105:0/3754210381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f7030101590 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 crypto rx=0x7f70280048d0 tx=0x7f7028009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 -- 172.21.15.105:0/3754210381 shutdown_connections 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 --2- 172.21.15.105:0/3754210381 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 0x7f70300fb1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 --2- 172.21.15.105:0/3754210381 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f7030101590 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 -- 172.21.15.105:0/3754210381 >> 172.21.15.105:0/3754210381 conn(0x7f70300f6db0 msgr2=0x7f70300f91c0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:38.098 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 -- 172.21.15.105:0/3754210381 shutdown_connections 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.095+0000 7f703750d640 1 -- 172.21.15.105:0/3754210381 wait complete. 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f703750d640 1 Processor -- start 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f703750d640 1 -- start start 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f703750d640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f703010df40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f703750d640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 0x7f703010e480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f703750d640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703010eb10 con 0x7f70301011c0 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f703750d640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703010ec80 con 0x7f7030101b60 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f70352a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f703010df40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f70352a9640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f703010df40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:36478/0 (socket says 172.21.15.105:36478) 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f70352a9640 1 -- 172.21.15.105:0/406655277 learned_addr learned my addr 172.21.15.105:0/406655277 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f7034aa8640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 0x7f703010e480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f70352a9640 1 -- 172.21.15.105:0/406655277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 msgr2=0x7f703010e480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:38.099 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f70352a9640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 0x7f703010e480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f70352a9640 1 -- 172.21.15.105:0/406655277 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7028005040 con 0x7f70301011c0 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.096+0000 7f7034aa8640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 0x7f703010e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).send_auth_request state changed! 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.097+0000 7f70352a9640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f703010df40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f70280048a0 tx=0x7f7028009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.097+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7028038070 con 0x7f70301011c0 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.097+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7028032d30 con 0x7f70301011c0 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.097+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7030116fb0 con 0x7f70301011c0 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.097+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f7028009680 con 0x7f70301011c0 2024-06-15T00:58:38.100 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.097+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f70301173f0 con 0x7f70301011c0 2024-06-15T00:58:38.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.098+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70300fc930 con 0x7f70301011c0 2024-06-15T00:58:38.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.098+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f7028030020 con 0x7f70301011c0 2024-06-15T00:58:38.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.099+0000 7f702dffb640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7018065430 0x7f70180678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:38.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.099+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f70280a22b0 con 0x7f70301011c0 2024-06-15T00:58:38.101 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.099+0000 7f7034aa8640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7018065430 0x7f70180678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:38.102 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.099+0000 7f7034aa8640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7018065430 0x7f70180678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7024006fd0 tx=0x7f7024008040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:38.103 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.103+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f7028070760 con 0x7f70301011c0 2024-06-15T00:58:38.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.299+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f70300fa9b0 con 0x7f70301011c0 2024-06-15T00:58:38.300 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.300+0000 7f702dffb640 1 -- 172.21.15.105:0/406655277 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f7028073d80 con 0x7f70301011c0 2024-06-15T00:58:38.301 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:38.301 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:38.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7018065430 msgr2=0x7f70180678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:38.302 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7018065430 0x7f70180678e0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 crypto rx=0x7f7024006fd0 tx=0x7f7024008040 comp rx=0 tx=0).stop 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 msgr2=0x7f703010df40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f703010df40 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 crypto rx=0x7f70280048a0 tx=0x7f7028009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 shutdown_connections 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f7018065430 0x7f70180678e0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f7030101b60 0x7f703010e480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 --2- 172.21.15.105:0/406655277 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f70301011c0 0x7f703010df40 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 >> 172.21.15.105:0/406655277 conn(0x7f70300f6db0 msgr2=0x7f70300f7d80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 shutdown_connections 2024-06-15T00:58:38.303 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:38.302+0000 7f703750d640 1 -- 172.21.15.105:0/406655277 wait complete. 2024-06-15T00:58:38.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:38 smithi143 ceph-mon[35860]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:38.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:38 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/406655277' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:38.745 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:38 smithi105 ceph-mon[28877]: pgmap v305: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:38.745 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:38 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/406655277' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:40.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:40 smithi143 ceph-mon[35860]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:40.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:40 smithi143 ceph-mon[35860]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:58:40.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:40 smithi105 ceph-mon[28877]: pgmap v306: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:40.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:40 smithi105 ceph-mon[28877]: from='mgr.14214 172.21.15.105:0/1850362059' entity='mgr.smithi105.oorfym' cmd=[{"prefix": "osd blocklist ls", "format": "json"}]: dispatch 2024-06-15T00:58:41.974 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:42.237 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:42.584 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:42 smithi105 ceph-mon[28877]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:42.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:42 smithi143 ceph-mon[35860]: pgmap v307: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:43.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 -- 172.21.15.105:0/2301326021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0650a0 msgr2=0x7efc1c065470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:43.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 --2- 172.21.15.105:0/2301326021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0650a0 0x7efc1c065470 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 crypto rx=0x7efc140048d0 tx=0x7efc14009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:43.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 -- 172.21.15.105:0/2301326021 shutdown_connections 2024-06-15T00:58:43.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 --2- 172.21.15.105:0/2301326021 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c0fe2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:43.853 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 --2- 172.21.15.105:0/2301326021 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0650a0 0x7efc1c065470 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 -- 172.21.15.105:0/2301326021 >> 172.21.15.105:0/2301326021 conn(0x7efc1c072110 msgr2=0x7efc1c072510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 -- 172.21.15.105:0/2301326021 shutdown_connections 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 -- 172.21.15.105:0/2301326021 wait complete. 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 Processor -- start 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 -- start start 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c0650a0 0x7efc1c112fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.851+0000 7efc23940640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c113500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc23940640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c113b90 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc23940640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc1c10cf40 con 0x7efc1c0650a0 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c113500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c113500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40766/0 (socket says 172.21.15.105:40766) 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 -- 172.21.15.105:0/2806122987 learned_addr learned my addr 172.21.15.105:0/2806122987 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:43.854 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc216dc640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c0650a0 0x7efc1c112fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 -- 172.21.15.105:0/2806122987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c0650a0 msgr2=0x7efc1c112fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c0650a0 0x7efc1c112fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 -- 172.21.15.105:0/2806122987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc14005040 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc20edb640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c113500 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efc1000b790 tx=0x7efc100091d0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc1000c700 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efc1000cd10 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc1c10d220 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.852+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7efc10013370 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.855 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.853+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc1c10d740 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.853+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc1c0fda50 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.854+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7efc10012070 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.854+0000 7efc19ffb640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc04065430 0x7efc040678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:43.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.854+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7efc10082030 con 0x7efc1c0fdeb0 2024-06-15T00:58:43.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.854+0000 7efc216dc640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc04065430 0x7efc040678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:43.856 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.855+0000 7efc216dc640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc04065430 0x7efc040678e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efc14004540 tx=0x7efc14035040 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:43.858 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:43.858+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7efc10050560 con 0x7efc1c0fdeb0 2024-06-15T00:58:44.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.057+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7efc1c10e660 con 0x7efc1c0fdeb0 2024-06-15T00:58:44.058 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.058+0000 7efc19ffb640 1 -- 172.21.15.105:0/2806122987 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7efc10053b80 con 0x7efc1c0fdeb0 2024-06-15T00:58:44.058 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:44.058 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:44.060 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc04065430 msgr2=0x7efc040678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc04065430 0x7efc040678e0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 crypto rx=0x7efc14004540 tx=0x7efc14035040 comp rx=0 tx=0).stop 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 msgr2=0x7efc1c113500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c113500 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 crypto rx=0x7efc1000b790 tx=0x7efc100091d0 comp rx=0 tx=0).stop 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 shutdown_connections 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7efc04065430 0x7efc040678e0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7efc1c0fdeb0 0x7efc1c113500 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 --2- 172.21.15.105:0/2806122987 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7efc1c0650a0 0x7efc1c112fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 >> 172.21.15.105:0/2806122987 conn(0x7efc1c072110 msgr2=0x7efc1c0f8680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 shutdown_connections 2024-06-15T00:58:44.061 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:44.060+0000 7efc23940640 1 -- 172.21.15.105:0/2806122987 wait complete. 2024-06-15T00:58:44.588 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:44 smithi105 ceph-mon[28877]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:44.588 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:44 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2806122987' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:44.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:44 smithi143 ceph-mon[35860]: pgmap v308: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:44.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:44 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2806122987' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:46.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:46 smithi143 ceph-mon[35860]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:46.777 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:46 smithi105 ceph-mon[28877]: pgmap v309: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:47.614 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm --image quay.ceph.io/ceph-ci/ceph:quincy shell --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 -- ceph health --format=json 2024-06-15T00:58:47.872 INFO:teuthology.orchestra.run.smithi105.stderr:Inferring config /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/mon.smithi105/config 2024-06-15T00:58:48.527 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:48 smithi105 ceph-mon[28877]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:48.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:48 smithi143 ceph-mon[35860]: pgmap v310: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:49.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.281+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2541926503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 msgr2=0x7f1c080fb1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:49.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.281+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2541926503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c080fb1e0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80048d0 tx=0x7f1bf8009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2541926503 shutdown_connections 2024-06-15T00:58:49.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2541926503 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c080fb1e0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.283 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2541926503 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c081011b0 0x7f1c08101580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2541926503 >> 172.21.15.105:0/2541926503 conn(0x7f1c080f6d80 msgr2=0x7f1c080f9190 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2541926503 shutdown_connections 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2541926503 wait complete. 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 Processor -- start 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- start start 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 --2- >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c081011b0 0x7f1c08112fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c081134e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c08113b70 con 0x7f1c08101b50 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.282+0000 7f1c0fbf6640 1 -- --> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c0810cf60 con 0x7f1c081011b0 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c081134e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 --2- >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c081134e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).handle_hello peer v2:172.21.15.105:3300/0 says I am v2:172.21.15.105:40782/0 (socket says 172.21.15.105:40782) 2024-06-15T00:58:49.284 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 -- 172.21.15.105:0/2101775359 learned_addr learned my addr 172.21.15.105:0/2101775359 (peer_addr_for_me v2:172.21.15.105:0/0) 2024-06-15T00:58:49.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d992640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c081011b0 0x7f1c08112fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:49.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 -- 172.21.15.105:0/2101775359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c081011b0 msgr2=0x7f1c08112fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:49.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c081011b0 0x7f1c08112fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 -- 172.21.15.105:0/2101775359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf8005040 con 0x7f1c08101b50 2024-06-15T00:58:49.285 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0d191640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c081134e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80048a0 tx=0x7f1bf8009da0 comp rx=0 tx=0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:49.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 1 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bf8038070 con 0x7f1c08101b50 2024-06-15T00:58:49.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 2 ==== config(26 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bf8032d30 con 0x7f1c08101b50 2024-06-15T00:58:49.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 3 ==== mon_map magic: 0 v1 ==== 357+0+0 (secure 0 0 0) 0x7f1bf8009680 con 0x7f1c08101b50 2024-06-15T00:58:49.286 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c0810d1e0 con 0x7f1c08101b50 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.283+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c0810d6a0 con 0x7f1c08101b50 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.284+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c080fc920 con 0x7f1c08101b50 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.285+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 4 ==== mgrmap(e 19) v1 ==== 83747+0+0 (secure 0 0 0) 0x7f1bf8030020 con 0x7f1c08101b50 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.285+0000 7f1bfe7fc640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1bf4065430 0x7f1bf40678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0).connect 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.285+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 4634+0+0 (secure 0 0 0) 0x7f1bf80a21b0 con 0x7f1c08101b50 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.285+0000 7f1c0d992640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1bf4065430 0x7f1bf40678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 crypto rx=0 tx=0 comp rx=0 tx=0)._handle_peer_banner_payload supported=3 required=0 2024-06-15T00:58:49.287 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.285+0000 7f1c0d992640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1bf4065430 0x7f1bf40678e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009b50 tx=0x7f1c04009120 comp rx=0 tx=0).ready entity=mgr.14214 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-06-15T00:58:49.289 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.289+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+175578 (secure 0 0 0) 0x7f1bf8070710 con 0x7f1c08101b50 2024-06-15T00:58:49.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.486+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 --> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7f1c08108230 con 0x7f1c08101b50 2024-06-15T00:58:49.486 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.486+0000 7f1bfe7fc640 1 -- 172.21.15.105:0/2101775359 <== mon.0 v2:172.21.15.105:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+184 (secure 0 0 0) 0x7f1bf8073d30 con 0x7f1c08101b50 2024-06-15T00:58:49.487 INFO:teuthology.orchestra.run.smithi105.stdout: 2024-06-15T00:58:49.487 INFO:teuthology.orchestra.run.smithi105.stdout:{"status":"HEALTH_WARN","checks":{"RECENT_MGR_MODULE_CRASH":{"severity":"HEALTH_WARN","summary":{"message":"3 mgr modules have recently crashed","count":3},"muted":false}},"mutes":[]} 2024-06-15T00:58:49.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1bf4065430 msgr2=0x7f1bf40678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:49.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1bf4065430 0x7f1bf40678e0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 crypto rx=0x7f1c04009b50 tx=0x7f1c04009120 comp rx=0 tx=0).stop 2024-06-15T00:58:49.488 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 msgr2=0x7f1c081134e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c081134e0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 crypto rx=0x7f1bf80048a0 tx=0x7f1bf8009da0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 shutdown_connections 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:6800/3392458267,v1:172.21.15.105:6801/3392458267] conn(0x7f1bf4065430 0x7f1bf40678e0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.105:3300/0,v1:172.21.15.105:6789/0] conn(0x7f1c08101b50 0x7f1c081134e0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 --2- 172.21.15.105:0/2101775359 >> [v2:172.21.15.143:3300/0,v1:172.21.15.143:6789/0] conn(0x7f1c081011b0 0x7f1c08112fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 crypto rx=0 tx=0 comp rx=0 tx=0).stop 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 >> 172.21.15.105:0/2101775359 conn(0x7f1c080f6d80 msgr2=0x7f1c08065790 unknown :-1 s=STATE_NONE l=0).mark_down 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 shutdown_connections 2024-06-15T00:58:49.489 INFO:teuthology.orchestra.run.smithi105.stderr:2024-06-15T00:58:49.488+0000 7f1c0fbf6640 1 -- 172.21.15.105:0/2101775359 wait complete. 2024-06-15T00:58:50.170 INFO:tasks.cephadm:Teardown begin 2024-06-15T00:58:50.170 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/contextutil.py", line 32, in nested yield vars File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/cephadm.py", line 1747, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph.py", line 1494, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph_manager.py", line 3191, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-15T00:58:50.173 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:58:50.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:58:50.234 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-06-15T00:58:50.234 DEBUG:teuthology.orchestra.run.smithi105:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-15T00:58:50.257 DEBUG:teuthology.orchestra.run.smithi143:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-06-15T00:58:50.292 INFO:tasks.cephadm:Stopping all daemons... 2024-06-15T00:58:50.292 INFO:tasks.cephadm.mon.smithi105:Stopping mon.smithi105... 2024-06-15T00:58:50.292 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105 2024-06-15T00:58:50.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:50 smithi105 ceph-mon[28877]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:50.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:50 smithi105 ceph-mon[28877]: from='client.? 172.21.15.105:0/2101775359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:50.598 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:50 smithi105 systemd[1]: Stopping Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:58:50.635 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:50 smithi143 ceph-mon[35860]: pgmap v311: 1 pgs: 1 active+clean; 577 KiB data, 1.7 GiB used, 535 GiB / 536 GiB avail 2024-06-15T00:58:50.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:50 smithi143 ceph-mon[35860]: from='client.? 172.21.15.105:0/2101775359' entity='client.admin' cmd=[{"prefix": "health", "format": "json"}]: dispatch 2024-06-15T00:58:50.989 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:50 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105[28873]: 2024-06-15T00:58:50.596+0000 7f0a4524f640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi105 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:58:50.990 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:50 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105[28873]: 2024-06-15T00:58:50.596+0000 7f0a4524f640 -1 mon.smithi105@0(leader) e2 *** Got Signal Terminated *** 2024-06-15T00:58:50.990 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:50 smithi105 podman[79610]: 2024-06-15 00:58:50.759668698 +0000 UTC m=+0.275772948 container died 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0) 2024-06-15T00:58:51.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:51 smithi105 podman[79610]: 2024-06-15 00:58:51.164036493 +0000 UTC m=+0.680140742 container cleanup 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True, org.label-schema.build-date=20240612, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux ) 2024-06-15T00:58:51.277 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:51 smithi105 bash[79610]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105 2024-06-15T00:58:51.593 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service' 2024-06-15T00:58:51.615 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:51 smithi105 podman[79690]: 2024-06-15 00:58:51.349177342 +0000 UTC m=+0.593493680 container remove 696d277ecc133d087d22beff0657e4d7d07260bf94a4517375a5a504e2eb27eb (image=quay.ceph.io/ceph-ci/ceph:quincy, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi105, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS) 2024-06-15T00:58:51.615 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:51 smithi105 systemd[1]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service: Deactivated successfully. 2024-06-15T00:58:51.615 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:51 smithi105 systemd[1]: Stopped Ceph mon.smithi105 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968. 2024-06-15T00:58:51.615 INFO:journalctl@ceph.mon.smithi105.smithi105.stdout:Jun 15 00:58:51 smithi105 systemd[1]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi105.service: Consumed 8.020s CPU time. 2024-06-15T00:58:51.639 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:58:51.639 INFO:tasks.cephadm.mon.smithi105:Stopped mon.smithi105 2024-06-15T00:58:51.640 INFO:tasks.cephadm.mon.smithi143:Stopping mon.smithi143... 2024-06-15T00:58:51.640 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi143 2024-06-15T00:58:52.043 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:51 smithi143 systemd[1]: Stopping Ceph mon.smithi143 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:58:52.359 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:52 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143[35856]: 2024-06-15T00:58:52.043+0000 7fe1ec630640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi143 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-journald=true --default-mon-cluster-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:58:52.360 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:52 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143[35856]: 2024-06-15T00:58:52.043+0000 7fe1ec630640 -1 mon.smithi143@1(peon) e2 *** Got Signal Terminated *** 2024-06-15T00:58:52.360 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:52 smithi143 podman[59425]: 2024-06-15 00:58:52.094812085 +0000 UTC m=+0.248321031 container died 54e49738eb43ccb7b97d6e089acfa52a9966c6436c41a6ec8f1876ca9ffa08cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=) 2024-06-15T00:58:52.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:52 smithi143 podman[59425]: 2024-06-15 00:58:52.568012253 +0000 UTC m=+0.721521199 container cleanup 54e49738eb43ccb7b97d6e089acfa52a9966c6436c41a6ec8f1876ca9ffa08cf (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.schema-version=1.0) 2024-06-15T00:58:52.636 INFO:journalctl@ceph.mon.smithi143.smithi143.stdout:Jun 15 00:58:52 smithi143 bash[59425]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-mon-smithi143 2024-06-15T00:58:53.006 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@mon.smithi143.service' 2024-06-15T00:58:53.048 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:58:53.049 INFO:tasks.cephadm.mon.smithi143:Stopped mon.smithi143 2024-06-15T00:58:53.049 INFO:tasks.cephadm.osd.0:Stopping osd.0... 2024-06-15T00:58:53.049 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.0 2024-06-15T00:58:53.343 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:53 smithi105 systemd[1]: Stopping Ceph osd.0 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:58:53.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:53 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0[46846]: 2024-06-15T00:58:53.343+0000 7f5ba16de640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.0 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:58:53.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:53 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0[46846]: 2024-06-15T00:58:53.343+0000 7f5ba16de640 -1 osd.0 41 *** Got signal Terminated *** 2024-06-15T00:58:53.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:53 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0[46846]: 2024-06-15T00:58:53.343+0000 7f5ba16de640 -1 osd.0 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:58:59.027 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:58 smithi105 podman[79829]: 2024-06-15 00:58:58.74846863 +0000 UTC m=+5.500441091 container died 2e48e6031e600d5d5755f5f33ca88149fe6b0ff33bf3121df0226b777d4aec75 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b) 2024-06-15T00:58:59.470 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:59 smithi105 podman[79829]: 2024-06-15 00:58:59.269176429 +0000 UTC m=+6.021148888 container cleanup 2e48e6031e600d5d5755f5f33ca88149fe6b0ff33bf3121df0226b777d4aec75 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0) 2024-06-15T00:58:59.470 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:59 smithi105 bash[79829]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0 2024-06-15T00:58:59.777 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:59 smithi105 podman[79844]: 2024-06-15 00:58:59.470632478 +0000 UTC m=+0.720193929 container remove 2e48e6031e600d5d5755f5f33ca88149fe6b0ff33bf3121df0226b777d4aec75 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, ceph=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:59:00.165 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:59 smithi105 podman[79933]: 2024-06-15 00:58:59.79736238 +0000 UTC m=+0.017150993 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:59:00.165 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:58:59 smithi105 podman[79933]: 2024-06-15 00:58:59.921505232 +0000 UTC m=+0.141293844 container create ab9e9a7d0bc85caded315f8d500225f912fd35c882cf3a6b2ae610cdd0c14c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.license=GPLv2, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux ) 2024-06-15T00:59:00.165 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 podman[79933]: 2024-06-15 00:59:00.165856992 +0000 UTC m=+0.385645608 container init ab9e9a7d0bc85caded315f8d500225f912fd35c882cf3a6b2ae610cdd0c14c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-15T00:59:00.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 podman[79933]: 2024-06-15 00:59:00.169514473 +0000 UTC m=+0.389303093 container start ab9e9a7d0bc85caded315f8d500225f912fd35c882cf3a6b2ae610cdd0c14c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, ceph=True, org.label-schema.build-date=20240612, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-15T00:59:00.426 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 podman[79933]: 2024-06-15 00:59:00.224279651 +0000 UTC m=+0.444068279 container attach ab9e9a7d0bc85caded315f8d500225f912fd35c882cf3a6b2ae610cdd0c14c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS) 2024-06-15T00:59:00.702 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 podman[79933]: 2024-06-15 00:59:00.426172869 +0000 UTC m=+0.645961490 container died ab9e9a7d0bc85caded315f8d500225f912fd35c882cf3a6b2ae610cdd0c14c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0-deactivate, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, ceph=True, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:59:00.952 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.0.service' 2024-06-15T00:59:00.975 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 podman[79957]: 2024-06-15 00:59:00.936180287 +0000 UTC m=+0.498203334 container remove ab9e9a7d0bc85caded315f8d500225f912fd35c882cf3a6b2ae610cdd0c14c9b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-0-deactivate, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux ) 2024-06-15T00:59:00.976 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 systemd[1]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.0.service: Deactivated successfully. 2024-06-15T00:59:00.976 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 systemd[1]: Stopped Ceph osd.0 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968. 2024-06-15T00:59:00.976 INFO:journalctl@ceph.osd.0.smithi105.stdout:Jun 15 00:59:00 smithi105 systemd[1]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.0.service: Consumed 3.567s CPU time. 2024-06-15T00:59:00.992 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:59:00.992 INFO:tasks.cephadm.osd.0:Stopped osd.0 2024-06-15T00:59:00.993 INFO:tasks.cephadm.osd.1:Stopping osd.1... 2024-06-15T00:59:00.993 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.1 2024-06-15T00:59:01.277 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:01 smithi105 systemd[1]: Stopping Ceph osd.1 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:59:01.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:01 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1[54509]: 2024-06-15T00:59:01.313+0000 7f9024c95640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.1 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:59:01.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:01 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1[54509]: 2024-06-15T00:59:01.313+0000 7f9024c95640 -1 osd.1 41 *** Got signal Terminated *** 2024-06-15T00:59:01.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:01 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1[54509]: 2024-06-15T00:59:01.313+0000 7f9024c95640 -1 osd.1 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:59:07.091 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:06 smithi105 podman[80047]: 2024-06-15 00:59:06.802995659 +0000 UTC m=+5.580636532 container died 8a821c8fbb80aa4ca65bbbd7bd99c19c9c3bf71a34435354c074b82b6cb9def3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, ceph=True, GIT_BRANCH=HEAD, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0) 2024-06-15T00:59:07.415 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:07 smithi105 podman[80047]: 2024-06-15 00:59:07.239517451 +0000 UTC m=+6.017158317 container cleanup 8a821c8fbb80aa4ca65bbbd7bd99c19c9c3bf71a34435354c074b82b6cb9def3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1, ceph=True, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-15T00:59:07.416 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:07 smithi105 bash[80047]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1 2024-06-15T00:59:07.777 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:07 smithi105 podman[80063]: 2024-06-15 00:59:07.415916885 +0000 UTC m=+0.608363883 container remove 8a821c8fbb80aa4ca65bbbd7bd99c19c9c3bf71a34435354c074b82b6cb9def3 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, GIT_CLEAN=True, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, io.buildah.version=1.36.0) 2024-06-15T00:59:08.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:07 smithi105 podman[80152]: 2024-06-15 00:59:07.771843692 +0000 UTC m=+0.017059721 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:59:08.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:07 smithi105 podman[80152]: 2024-06-15 00:59:07.874921368 +0000 UTC m=+0.120137382 container create 9333c36c5a5e5185bfa02fd9e7fd29681eaa7f114eebe6555fb93089bfd8c5f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_CLEAN=True) 2024-06-15T00:59:08.168 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:08 smithi105 podman[80152]: 2024-06-15 00:59:08.0944589 +0000 UTC m=+0.339674929 container init 9333c36c5a5e5185bfa02fd9e7fd29681eaa7f114eebe6555fb93089bfd8c5f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux ) 2024-06-15T00:59:08.169 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:08 smithi105 podman[80152]: 2024-06-15 00:59:08.0981146 +0000 UTC m=+0.343330616 container start 9333c36c5a5e5185bfa02fd9e7fd29681eaa7f114eebe6555fb93089bfd8c5f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, ceph=True) 2024-06-15T00:59:08.169 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:08 smithi105 podman[80152]: 2024-06-15 00:59:08.168911914 +0000 UTC m=+0.414127930 container attach 9333c36c5a5e5185bfa02fd9e7fd29681eaa7f114eebe6555fb93089bfd8c5f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1-deactivate, org.label-schema.schema-version=1.0, ceph=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True) 2024-06-15T00:59:08.527 INFO:journalctl@ceph.osd.1.smithi105.stdout:Jun 15 00:59:08 smithi105 podman[80152]: 2024-06-15 00:59:08.349521049 +0000 UTC m=+0.594737065 container died 9333c36c5a5e5185bfa02fd9e7fd29681eaa7f114eebe6555fb93089bfd8c5f9 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-1-deactivate, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux ) 2024-06-15T00:59:09.106 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.1.service' 2024-06-15T00:59:09.147 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:59:09.148 INFO:tasks.cephadm.osd.1:Stopped osd.1 2024-06-15T00:59:09.148 INFO:tasks.cephadm.osd.2:Stopping osd.2... 2024-06-15T00:59:09.148 DEBUG:teuthology.orchestra.run.smithi105:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.2 2024-06-15T00:59:09.493 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:09 smithi105 systemd[1]: Stopping Ceph osd.2 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:59:09.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:09 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2[62195]: 2024-06-15T00:59:09.492+0000 7f4415ab5640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.2 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:59:09.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:09 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2[62195]: 2024-06-15T00:59:09.492+0000 7f4415ab5640 -1 osd.2 41 *** Got signal Terminated *** 2024-06-15T00:59:09.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:09 smithi105 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2[62195]: 2024-06-15T00:59:09.492+0000 7f4415ab5640 -1 osd.2 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:59:15.118 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:14 smithi105 podman[80257]: 2024-06-15 00:59:14.844862281 +0000 UTC m=+5.458896172 container died dd2a1ae81f29544f925f2081f1635012c4353f0a9ee51c24722c9f8cb5ae3f65 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, ceph=True, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2) 2024-06-15T00:59:15.477 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:15 smithi105 podman[80257]: 2024-06-15 00:59:15.284314456 +0000 UTC m=+5.898348375 container cleanup dd2a1ae81f29544f925f2081f1635012c4353f0a9ee51c24722c9f8cb5ae3f65 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_CLEAN=True) 2024-06-15T00:59:15.477 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:15 smithi105 bash[80257]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2 2024-06-15T00:59:15.777 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:15 smithi105 podman[80274]: 2024-06-15 00:59:15.477281692 +0000 UTC m=+0.632467077 container remove dd2a1ae81f29544f925f2081f1635012c4353f0a9ee51c24722c9f8cb5ae3f65 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, org.label-schema.schema-version=1.0, ceph=True) 2024-06-15T00:59:16.230 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:15 smithi105 podman[80365]: 2024-06-15 00:59:15.808304337 +0000 UTC m=+0.016798720 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:59:16.230 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:15 smithi105 podman[80365]: 2024-06-15 00:59:15.936485686 +0000 UTC m=+0.144980065 container create cacc3a4ec1a69c1e45ca03c994f5344c72c7c716741a88be7c0782110b61b7f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2-deactivate, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, ceph=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_CLEAN=True, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612) 2024-06-15T00:59:16.230 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:16 smithi105 podman[80365]: 2024-06-15 00:59:16.147399566 +0000 UTC m=+0.355893955 container init cacc3a4ec1a69c1e45ca03c994f5344c72c7c716741a88be7c0782110b61b7f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, ceph=True, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:59:16.230 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:16 smithi105 podman[80365]: 2024-06-15 00:59:16.150897505 +0000 UTC m=+0.359391877 container start cacc3a4ec1a69c1e45ca03c994f5344c72c7c716741a88be7c0782110b61b7f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:59:16.231 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:16 smithi105 podman[80365]: 2024-06-15 00:59:16.230709341 +0000 UTC m=+0.439203719 container attach cacc3a4ec1a69c1e45ca03c994f5344c72c7c716741a88be7c0782110b61b7f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2-deactivate, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, maintainer=Guillaume Abrioux , io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, ceph=True, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0) 2024-06-15T00:59:16.527 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:16 smithi105 podman[80365]: 2024-06-15 00:59:16.398155336 +0000 UTC m=+0.606649710 container died cacc3a4ec1a69c1e45ca03c994f5344c72c7c716741a88be7c0782110b61b7f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2-deactivate, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240612, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.license=GPLv2, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, RELEASE=quincy-e47ff4b) 2024-06-15T00:59:16.929 INFO:journalctl@ceph.osd.2.smithi105.stdout:Jun 15 00:59:16 smithi105 podman[80388]: 2024-06-15 00:59:16.909092177 +0000 UTC m=+0.501619344 container remove cacc3a4ec1a69c1e45ca03c994f5344c72c7c716741a88be7c0782110b61b7f5 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-2-deactivate, org.label-schema.vendor=CentOS, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0) 2024-06-15T00:59:16.935 DEBUG:teuthology.orchestra.run.smithi105:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.2.service' 2024-06-15T00:59:16.972 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:59:16.973 INFO:tasks.cephadm.osd.2:Stopped osd.2 2024-06-15T00:59:16.973 INFO:tasks.cephadm.osd.3:Stopping osd.3... 2024-06-15T00:59:16.973 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.3 2024-06-15T00:59:17.278 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:17 smithi143 systemd[1]: Stopping Ceph osd.3 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:59:17.636 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:17 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3[41369]: 2024-06-15T00:59:17.277+0000 7fc45894a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.3 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:59:17.636 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:17 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3[41369]: 2024-06-15T00:59:17.277+0000 7fc45894a640 -1 osd.3 41 *** Got signal Terminated *** 2024-06-15T00:59:17.636 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:17 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3[41369]: 2024-06-15T00:59:17.277+0000 7fc45894a640 -1 osd.3 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:59:21.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:20 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:20.753+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:22.136 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:21 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3[41369]: 2024-06-15T00:59:21.731+0000 7fc452f5e640 -1 osd.3 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:55.519746+0000 front 2024-06-15T00:58:55.519800+0000 (oldest deadline 2024-06-15T00:59:20.819552+0000) 2024-06-15T00:59:22.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:21 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:21.792+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:23.120 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:22 smithi143 podman[59558]: 2024-06-15 00:59:22.82179587 +0000 UTC m=+5.651415112 container died 963d01b8462b5729df12c8b0e6db45476979a4f0f1c5d324698e70ca0d2a620b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612) 2024-06-15T00:59:23.121 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:22 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:22.814+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:23.386 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 podman[59558]: 2024-06-15 00:59:23.194221896 +0000 UTC m=+6.023841140 container cleanup 963d01b8462b5729df12c8b0e6db45476979a4f0f1c5d324698e70ca0d2a620b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2, GIT_CLEAN=True, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, RELEASE=quincy-e47ff4b, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47) 2024-06-15T00:59:23.386 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 bash[59558]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3 2024-06-15T00:59:23.386 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 podman[59574]: 2024-06-15 00:59:23.354555758 +0000 UTC m=+0.532235108 container remove 963d01b8462b5729df12c8b0e6db45476979a4f0f1c5d324698e70ca0d2a620b (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.build-date=20240612, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, CEPH_POINT_RELEASE=, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, org.label-schema.license=GPLv2) 2024-06-15T00:59:24.041 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:23 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:23.731+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:24.041 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:23 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:23.825+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:24.042 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 podman[59663]: 2024-06-15 00:59:23.76353794 +0000 UTC m=+0.109972496 container create d1dae1b9c0606ad55bb1de70f5066a7241479d3a0cfbda8b35be0526d71e67ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, ceph=True, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:59:24.042 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 podman[59663]: 2024-06-15 00:59:23.670837727 +0000 UTC m=+0.017272294 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:59:24.042 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 podman[59663]: 2024-06-15 00:59:23.96666321 +0000 UTC m=+0.313097782 container init d1dae1b9c0606ad55bb1de70f5066a7241479d3a0cfbda8b35be0526d71e67ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-15T00:59:24.042 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:23 smithi143 podman[59663]: 2024-06-15 00:59:23.969927589 +0000 UTC m=+0.316362143 container start d1dae1b9c0606ad55bb1de70f5066a7241479d3a0cfbda8b35be0526d71e67ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3-deactivate, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, CEPH_POINT_RELEASE=, GIT_BRANCH=HEAD, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS) 2024-06-15T00:59:24.042 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:24 smithi143 podman[59663]: 2024-06-15 00:59:24.041553143 +0000 UTC m=+0.387987722 container attach d1dae1b9c0606ad55bb1de70f5066a7241479d3a0cfbda8b35be0526d71e67ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.schema-version=1.0, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, GIT_CLEAN=True, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, org.label-schema.license=GPLv2) 2024-06-15T00:59:24.386 INFO:journalctl@ceph.osd.3.smithi143.stdout:Jun 15 00:59:24 smithi143 podman[59663]: 2024-06-15 00:59:24.21774731 +0000 UTC m=+0.564181880 container died d1dae1b9c0606ad55bb1de70f5066a7241479d3a0cfbda8b35be0526d71e67ac (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-3-deactivate, org.label-schema.license=GPLv2, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , ceph=True, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b) 2024-06-15T00:59:24.801 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:24 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:24.732+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:24.801 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:24 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:24.801+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:24.830 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.3.service' 2024-06-15T00:59:24.868 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:59:24.868 INFO:tasks.cephadm.osd.3:Stopped osd.3 2024-06-15T00:59:24.868 INFO:tasks.cephadm.osd.4:Stopping osd.4... 2024-06-15T00:59:24.868 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.4 2024-06-15T00:59:25.135 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:24 smithi143 systemd[1]: Stopping Ceph osd.4 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:59:25.635 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:25 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:25.221+0000 7ffa4b344640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.4 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:59:25.636 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:25 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:25.221+0000 7ffa4b344640 -1 osd.4 41 *** Got signal Terminated *** 2024-06-15T00:59:25.636 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:25 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:25.221+0000 7ffa4b344640 -1 osd.4 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:59:26.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:25 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:25.758+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:26.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:25 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:25.755+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:27.124 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:26 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:26.728+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:27.124 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:26 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:26.741+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:28.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:27 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:27.763+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:28.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:27 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:27.770+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:29.135 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:28 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:28.741+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:29.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:28 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:28.771+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:29.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:28 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:28.771+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:06.263003+0000 front 2024-06-15T00:59:06.263072+0000 (oldest deadline 2024-06-15T00:59:28.562938+0000) 2024-06-15T00:59:30.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:29 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:29.740+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:30.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:29 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:29.747+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:57.562318+0000 front 2024-06-15T00:58:57.562274+0000 (oldest deadline 2024-06-15T00:59:23.462083+0000) 2024-06-15T00:59:30.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:29 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4[47712]: 2024-06-15T00:59:29.747+0000 7ffa45958640 -1 osd.4 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:06.263003+0000 front 2024-06-15T00:59:06.263072+0000 (oldest deadline 2024-06-15T00:59:28.562938+0000) 2024-06-15T00:59:30.846 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:30 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:30.753+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:30.847 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:30 smithi143 podman[59767]: 2024-06-15 00:59:30.533066988 +0000 UTC m=+5.438191708 container died f7e66720bac007f0b1bd29204587792c14ac9acf3142bee061505d5a99cf8a1f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, ceph=True, GIT_CLEAN=True, maintainer=Guillaume Abrioux , RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.license=GPLv2) 2024-06-15T00:59:30.847 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:30 smithi143 podman[59767]: 2024-06-15 00:59:30.847189913 +0000 UTC m=+5.752314634 container cleanup f7e66720bac007f0b1bd29204587792c14ac9acf3142bee061505d5a99cf8a1f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, maintainer=Guillaume Abrioux , GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_CLEAN=True, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, org.label-schema.vendor=CentOS, ceph=True) 2024-06-15T00:59:31.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:30 smithi143 bash[59767]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4 2024-06-15T00:59:31.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59796]: 2024-06-15 00:59:31.007008643 +0000 UTC m=+0.471158400 container remove f7e66720bac007f0b1bd29204587792c14ac9acf3142bee061505d5a99cf8a1f (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4, org.label-schema.name=CentOS Stream 9 Base Image, ceph=True, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, GIT_CLEAN=True, org.label-schema.license=GPLv2, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=) 2024-06-15T00:59:31.692 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59882]: 2024-06-15 00:59:31.416203666 +0000 UTC m=+0.107499141 container create c95eb0bdc45ef7120984bf8cb68c7f76e8ca666c6e38740e05da61fbe6d38d2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, GIT_BRANCH=HEAD, org.label-schema.license=GPLv2, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, org.label-schema.name=CentOS Stream 9 Base Image, io.buildah.version=1.36.0, GIT_CLEAN=True, maintainer=Guillaume Abrioux , ceph=True) 2024-06-15T00:59:31.693 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59882]: 2024-06-15 00:59:31.325642049 +0000 UTC m=+0.016937484 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:59:31.693 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59882]: 2024-06-15 00:59:31.618278939 +0000 UTC m=+0.309574382 container init c95eb0bdc45ef7120984bf8cb68c7f76e8ca666c6e38740e05da61fbe6d38d2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, CEPH_POINT_RELEASE=, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2) 2024-06-15T00:59:31.693 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59882]: 2024-06-15 00:59:31.621770963 +0000 UTC m=+0.313066394 container start c95eb0bdc45ef7120984bf8cb68c7f76e8ca666c6e38740e05da61fbe6d38d2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4-deactivate, org.label-schema.name=CentOS Stream 9 Base Image, GIT_CLEAN=True, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, ceph=True, CEPH_POINT_RELEASE=, org.label-schema.schema-version=1.0, GIT_REPO=git@github.com:ceph/ceph-container.git, maintainer=Guillaume Abrioux , org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-15T00:59:31.693 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59882]: 2024-06-15 00:59:31.693181477 +0000 UTC m=+0.384476921 container attach c95eb0bdc45ef7120984bf8cb68c7f76e8ca666c6e38740e05da61fbe6d38d2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, io.buildah.version=1.36.0, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-15T00:59:32.136 INFO:journalctl@ceph.osd.4.smithi143.stdout:Jun 15 00:59:31 smithi143 podman[59882]: 2024-06-15 00:59:31.868037362 +0000 UTC m=+0.559332792 container died c95eb0bdc45ef7120984bf8cb68c7f76e8ca666c6e38740e05da61fbe6d38d2a (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-4-deactivate, GIT_CLEAN=True, org.label-schema.license=GPLv2, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, org.label-schema.schema-version=1.0, org.label-schema.name=CentOS Stream 9 Base Image, RELEASE=quincy-e47ff4b, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, org.label-schema.build-date=20240612) 2024-06-15T00:59:32.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:31 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:31.789+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:32.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:31 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:31.789+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:32.471 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.4.service' 2024-06-15T00:59:32.509 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:59:32.509 INFO:tasks.cephadm.osd.4:Stopped osd.4 2024-06-15T00:59:32.509 INFO:tasks.cephadm.osd.5:Stopping osd.5... 2024-06-15T00:59:32.509 DEBUG:teuthology.orchestra.run.smithi143:> sudo systemctl stop ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.5 2024-06-15T00:59:32.794 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:32 smithi143 systemd[1]: Stopping Ceph osd.5 for bd0fff6c-2ab0-11ef-bc9d-c7b262605968... 2024-06-15T00:59:33.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:32 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:32.794+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:33.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:32 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:32.794+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:33.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:32 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:32.872+0000 7fcf35b1a640 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-osd -n osd.5 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-journald=true --default-log-to-stderr=false (PID: 1) UID: 0 2024-06-15T00:59:33.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:32 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:32.872+0000 7fcf35b1a640 -1 osd.5 41 *** Got signal Terminated *** 2024-06-15T00:59:33.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:32 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:32.872+0000 7fcf35b1a640 -1 osd.5 41 *** Immediate shutdown (osd_fast_shutdown=true) *** 2024-06-15T00:59:34.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:33 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:33.771+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:34.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:33 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:33.771+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:35.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:34 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:34.809+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:35.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:34 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:34.809+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:36.135 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:35 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:35.803+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:36.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:35 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:35.803+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:37.135 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:36 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:36.782+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:37.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:36 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:36.782+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:38.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:37 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:37.814+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6806 osd.0 since back 2024-06-15T00:58:54.686955+0000 front 2024-06-15T00:58:54.687052+0000 (oldest deadline 2024-06-15T00:59:20.586895+0000) 2024-06-15T00:59:38.136 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:37 smithi143 ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5[54108]: 2024-06-15T00:59:37.814+0000 7fcf31931640 -1 osd.5 41 heartbeat_check: no reply from 172.21.15.105:6814 osd.1 since back 2024-06-15T00:59:05.887694+0000 front 2024-06-15T00:59:05.887668+0000 (oldest deadline 2024-06-15T00:59:31.787517+0000) 2024-06-15T00:59:38.624 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:38 smithi143 podman[59985]: 2024-06-15 00:59:38.310820552 +0000 UTC m=+5.578966683 container died 8d2c1b327d08aae0db8d19a0bd37ddbbedf70a2538c7fd80d62b9e7cb10cb6ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, CEPH_POINT_RELEASE=, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.schema-version=1.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, RELEASE=quincy-e47ff4b, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.build-date=20240612, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-15T00:59:38.625 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:38 smithi143 podman[59985]: 2024-06-15 00:59:38.624914883 +0000 UTC m=+5.893061010 container cleanup 8d2c1b327d08aae0db8d19a0bd37ddbbedf70a2538c7fd80d62b9e7cb10cb6ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5, io.buildah.version=1.36.0, GIT_BRANCH=HEAD, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.build-date=20240612, maintainer=Guillaume Abrioux , GIT_REPO=git@github.com:ceph/ceph-container.git, ceph=True, GIT_CLEAN=True, org.label-schema.license=GPLv2, org.label-schema.vendor=CentOS, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b) 2024-06-15T00:59:38.886 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:38 smithi143 bash[59985]: ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5 2024-06-15T00:59:38.886 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:38 smithi143 podman[60002]: 2024-06-15 00:59:38.817549842 +0000 UTC m=+0.505683936 container remove 8d2c1b327d08aae0db8d19a0bd37ddbbedf70a2538c7fd80d62b9e7cb10cb6ea (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, GIT_REPO=git@github.com:ceph/ceph-container.git, org.label-schema.vendor=CentOS, GIT_CLEAN=True, org.label-schema.schema-version=1.0, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.build-date=20240612, org.label-schema.name=CentOS Stream 9 Base Image, CEPH_POINT_RELEASE=, RELEASE=quincy-e47ff4b, io.buildah.version=1.36.0) 2024-06-15T00:59:39.495 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:39 smithi143 podman[60094]: 2024-06-15 00:59:39.226578175 +0000 UTC m=+0.091705914 container create d1c99d3e48ac7c9070aeb406d09d476368d507398d5c70f93920bc46a107e320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.vendor=CentOS, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, maintainer=Guillaume Abrioux , CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, io.buildah.version=1.36.0, org.label-schema.license=GPLv2, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True) 2024-06-15T00:59:39.495 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:39 smithi143 podman[60094]: 2024-06-15 00:59:39.15168953 +0000 UTC m=+0.016817239 image pull 37f9eedc23c32ee6d04590e8a81233336d2c14cd278f52a2a2746a97e6ef2d95 quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580 2024-06-15T00:59:39.495 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:39 smithi143 podman[60094]: 2024-06-15 00:59:39.420587441 +0000 UTC m=+0.285715148 container init d1c99d3e48ac7c9070aeb406d09d476368d507398d5c70f93920bc46a107e320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.vendor=CentOS, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , GIT_BRANCH=HEAD, org.label-schema.schema-version=1.0, RELEASE=quincy-e47ff4b, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, ceph=True, io.buildah.version=1.36.0) 2024-06-15T00:59:39.496 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:39 smithi143 podman[60094]: 2024-06-15 00:59:39.424090024 +0000 UTC m=+0.289217722 container start d1c99d3e48ac7c9070aeb406d09d476368d507398d5c70f93920bc46a107e320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5-deactivate, maintainer=Guillaume Abrioux , org.label-schema.name=CentOS Stream 9 Base Image, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, CEPH_POINT_RELEASE=, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_CLEAN=True, RELEASE=quincy-e47ff4b, org.label-schema.license=GPLv2, GIT_BRANCH=HEAD, ceph=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, org.label-schema.vendor=CentOS, io.buildah.version=1.36.0) 2024-06-15T00:59:39.496 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:39 smithi143 podman[60094]: 2024-06-15 00:59:39.495361725 +0000 UTC m=+0.360489442 container attach d1c99d3e48ac7c9070aeb406d09d476368d507398d5c70f93920bc46a107e320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , org.label-schema.schema-version=1.0, CEPH_POINT_RELEASE=, org.label-schema.vendor=CentOS, RELEASE=quincy-e47ff4b, org.label-schema.build-date=20240612, ceph=True, GIT_REPO=git@github.com:ceph/ceph-container.git, GIT_BRANCH=HEAD, GIT_CLEAN=True, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, io.buildah.version=1.36.0, org.label-schema.name=CentOS Stream 9 Base Image) 2024-06-15T00:59:39.886 INFO:journalctl@ceph.osd.5.smithi143.stdout:Jun 15 00:59:39 smithi143 podman[60094]: 2024-06-15 00:59:39.674567406 +0000 UTC m=+0.539695115 container died d1c99d3e48ac7c9070aeb406d09d476368d507398d5c70f93920bc46a107e320 (image=quay.ceph.io/ceph-ci/ceph@sha256:7fd30dfff71893ab9541e2e03f37cd56509c88398ea6d5de47fbc7efe7a87580, name=ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968-osd-5-deactivate, org.label-schema.schema-version=1.0, org.label-schema.build-date=20240612, org.label-schema.license=GPLv2, maintainer=Guillaume Abrioux , ceph=True, io.buildah.version=1.36.0, GIT_COMMIT=4a8012357b057f80e264ea5a6c36c04375a85d47, CEPH_POINT_RELEASE=, GIT_CLEAN=True, org.label-schema.name=CentOS Stream 9 Base Image, GIT_BRANCH=HEAD, RELEASE=quincy-e47ff4b, org.label-schema.vendor=CentOS, GIT_REPO=git@github.com:ceph/ceph-container.git) 2024-06-15T00:59:40.273 DEBUG:teuthology.orchestra.run.smithi143:> sudo pkill -f 'journalctl -f -n 0 -u ceph-bd0fff6c-2ab0-11ef-bc9d-c7b262605968@osd.5.service' 2024-06-15T00:59:40.310 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-06-15T00:59:40.311 INFO:tasks.cephadm.osd.5:Stopped osd.5 2024-06-15T00:59:40.311 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-15T00:59:49.285 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 --force --keep-logs 2024-06-15T00:59:54.168 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:59:54.201 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f /etc/ceph/ceph.conf /etc/ceph/ceph.client.admin.keyring 2024-06-15T00:59:54.228 INFO:tasks.cephadm:Archiving crash dumps... 2024-06-15T00:59:54.230 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/remote/smithi105/crash 2024-06-15T00:59:54.230 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/crash -- . 2024-06-15T00:59:54.313 INFO:teuthology.orchestra.run.smithi105.stderr:tar: /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-15T00:59:54.313 INFO:teuthology.orchestra.run.smithi105.stderr:tar: Error is not recoverable: exiting now 2024-06-15T00:59:54.315 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/crash to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/remote/smithi143/crash 2024-06-15T00:59:54.315 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/crash -- . 2024-06-15T00:59:54.351 INFO:teuthology.orchestra.run.smithi143.stderr:tar: /var/lib/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/crash: Cannot open: No such file or directory 2024-06-15T00:59:54.352 INFO:teuthology.orchestra.run.smithi143.stderr:tar: Error is not recoverable: exiting now 2024-06-15T00:59:54.353 INFO:tasks.cephadm:Checking cluster log for badness... 2024-06-15T00:59:54.353 DEBUG:teuthology.orchestra.run.smithi105:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v FS_DEGRADED | egrep -v FS_INLINE_DATA_DEPRECATED | egrep -v FS_WITH_FAILED_MDS | egrep -v MDS_ALL_DOWN | egrep -v MDS_DAMAGE | egrep -v MDS_DEGRADED | egrep -v MDS_FAILED | egrep -v MDS_INSUFFICIENT_STANDBY | egrep -v MDS_UP_LESS_THAN_MAX | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'overall HEALTH_' | egrep -v 'Replacing daemon' | egrep -v OSD_DOWN | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-06-15T00:59:54.403 INFO:teuthology.orchestra.run.smithi105.stderr:grep: /var/log/ceph/bd0fff6c-2ab0-11ef-bc9d-c7b262605968/ceph.log: No such file or directory 2024-06-15T00:59:54.406 INFO:tasks.cephadm:Compressing logs... 2024-06-15T00:59:54.406 DEBUG:teuthology.orchestra.run.smithi105:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:59:54.408 DEBUG:teuthology.orchestra.run.smithi143:> sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:59:54.431 INFO:teuthology.orchestra.run.smithi105.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-15T00:59:54.432 INFO:teuthology.orchestra.run.smithi143.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-06-15T00:59:55.129 INFO:tasks.cephadm:Archiving logs... 2024-06-15T00:59:55.129 DEBUG:teuthology.misc:Transferring archived files from smithi105:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/remote/smithi105/log 2024-06-15T00:59:55.130 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-15T00:59:55.285 DEBUG:teuthology.misc:Transferring archived files from smithi143:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/remote/smithi143/log 2024-06-15T00:59:55.286 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /var/log/ceph -- . 2024-06-15T00:59:55.359 INFO:tasks.cephadm:Removing cluster... 2024-06-15T00:59:55.359 DEBUG:teuthology.orchestra.run.smithi105:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 --force 2024-06-15T00:59:55.820 DEBUG:teuthology.orchestra.run.smithi143:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid bd0fff6c-2ab0-11ef-bc9d-c7b262605968 --force 2024-06-15T00:59:56.288 INFO:tasks.cephadm:Removing cephadm ... 2024-06-15T00:59:56.288 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-15T00:59:56.304 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf /home/ubuntu/cephtest/cephadm 2024-06-15T00:59:56.319 INFO:tasks.cephadm:Teardown complete 2024-06-15T00:59:56.320 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/cephadm.py", line 1747, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph.py", line 1494, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph_manager.py", line 3191, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-15T00:59:56.498 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=91ee14e767cb4b8fba5b92e6155f2632 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c1b06c2b5d3a02a3583265a115f09e052855bcf2/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.10/contextlib.py", line 135, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/cephadm.py", line 1747, in task healthy(ctx=ctx, config=config) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph.py", line 1494, in healthy manager.wait_until_healthy(timeout=300) File "/home/teuthworker/src/github.com_cbodley_ceph_e3daf20513a97cf7004cd9e7bfdba8f5a015acbc/qa/tasks/ceph_manager.py", line 3191, in wait_until_healthy assert time.time() - start < timeout, \ AssertionError: timeout expired in wait_until_healthy 2024-06-15T00:59:56.500 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-06-15T00:59:56.516 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-06-15T00:59:56.565 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-06-15T00:59:56.566 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T00:59:56.579 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-06-15T00:59:56.655 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-06-15T00:59:56.664 INFO:teuthology.task.clock:Checking final clock skew... 2024-06-15T00:59:56.664 DEBUG:teuthology.orchestra.run.smithi105:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T00:59:56.667 DEBUG:teuthology.orchestra.run.smithi143:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-06-15T00:59:56.680 INFO:teuthology.orchestra.run.smithi105.stderr:bash: line 1: ntpq: command not found 2024-06-15T00:59:56.681 INFO:teuthology.orchestra.run.smithi143.stderr:bash: line 1: ntpq: command not found 2024-06-15T00:59:56.685 INFO:teuthology.orchestra.run.smithi105.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-15T00:59:56.685 INFO:teuthology.orchestra.run.smithi105.stdout:=============================================================================== 2024-06-15T00:59:56.685 INFO:teuthology.orchestra.run.smithi105.stdout:^* hv01.front.sepia.ceph.com 3 6 377 19 +994us[ +970us] +/- 48ms 2024-06-15T00:59:56.685 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 22 -553us[ -577us] +/- 54ms 2024-06-15T00:59:56.685 INFO:teuthology.orchestra.run.smithi105.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 21 -1114us[-1138us] +/- 93ms 2024-06-15T00:59:56.685 INFO:teuthology.orchestra.run.smithi105.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:59:56.686 INFO:teuthology.orchestra.run.smithi143.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-06-15T00:59:56.686 INFO:teuthology.orchestra.run.smithi143.stdout:=============================================================================== 2024-06-15T00:59:56.687 INFO:teuthology.orchestra.run.smithi143.stdout:^* hv01.front.sepia.ceph.com 3 6 377 23 +973us[+1015us] +/- 48ms 2024-06-15T00:59:56.687 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv02.front.sepia.ceph.com 3 6 377 20 -572us[ -572us] +/- 54ms 2024-06-15T00:59:56.687 INFO:teuthology.orchestra.run.smithi143.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 20 -1126us[-1126us] +/- 93ms 2024-06-15T00:59:56.687 INFO:teuthology.orchestra.run.smithi143.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-06-15T00:59:56.687 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-06-15T00:59:56.695 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-06-15T00:59:56.695 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-06-15T00:59:56.714 DEBUG:teuthology.orchestra.run.smithi105:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-15T00:59:56.798 DEBUG:teuthology.orchestra.run.smithi143:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-06-15T00:59:56.863 DEBUG:teuthology.orchestra.run.smithi105:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-15T00:59:56.890 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:59:56.891 DEBUG:teuthology.orchestra.run.smithi143:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-06-15T00:59:56.917 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:59:56.918 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-06-15T00:59:56.927 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-06-15T00:59:56.949 INFO:teuthology.task.internal:Duration was 1194.205496 seconds 2024-06-15T00:59:56.950 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-06-15T00:59:56.983 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-06-15T00:59:56.983 DEBUG:teuthology.orchestra.run.smithi105:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T00:59:56.986 DEBUG:teuthology.orchestra.run.smithi143:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-06-15T00:59:57.022 INFO:teuthology.orchestra.run.smithi105.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-15T00:59:57.024 INFO:teuthology.orchestra.run.smithi143.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-06-15T00:59:57.426 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-06-15T00:59:57.426 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi105.front.sepia.ceph.com 2024-06-15T00:59:57.426 DEBUG:teuthology.orchestra.run.smithi105:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T00:59:57.446 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi143.front.sepia.ceph.com 2024-06-15T00:59:57.446 DEBUG:teuthology.orchestra.run.smithi143:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-06-15T00:59:57.487 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-06-15T00:59:57.488 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:59:57.490 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-06-15T00:59:57.554 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-06-15T00:59:57.554 DEBUG:teuthology.orchestra.run.smithi105:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T00:59:57.556 DEBUG:teuthology.orchestra.run.smithi143:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-06-15T00:59:57.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-06-15T00:59:57.948 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-06-15T00:59:57.949 DEBUG:teuthology.orchestra.run.smithi105:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T00:59:57.974 DEBUG:teuthology.orchestra.run.smithi143:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-06-15T00:59:57.999 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-06-15T00:59:58.027 DEBUG:teuthology.orchestra.run.smithi105:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:59:58.029 DEBUG:teuthology.orchestra.run.smithi143:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:59:58.050 INFO:teuthology.orchestra.run.smithi105.stdout:kernel.core_pattern = core 2024-06-15T00:59:58.063 INFO:teuthology.orchestra.run.smithi143.stdout:kernel.core_pattern = core 2024-06-15T00:59:58.082 DEBUG:teuthology.orchestra.run.smithi105:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:59:58.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:59:58.124 DEBUG:teuthology.orchestra.run.smithi143:> test -e /home/ubuntu/cephtest/archive/coredump 2024-06-15T00:59:58.138 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-06-15T00:59:58.139 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-06-15T00:59:58.148 INFO:teuthology.task.internal:Transferring archived files... 2024-06-15T00:59:58.149 DEBUG:teuthology.misc:Transferring archived files from smithi105:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/remote/smithi105 2024-06-15T00:59:58.149 DEBUG:teuthology.orchestra.run.smithi105:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T00:59:58.204 DEBUG:teuthology.misc:Transferring archived files from smithi143:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-06-14_18:29:32-orch-wip-yuri5-testing-2024-06-08-0902-reef-distro-default-smithi/7756127/remote/smithi143 2024-06-15T00:59:58.204 DEBUG:teuthology.orchestra.run.smithi143:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-06-15T00:59:58.238 INFO:teuthology.task.internal:Removing archive directory... 2024-06-15T00:59:58.239 DEBUG:teuthology.orchestra.run.smithi105:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T00:59:58.241 DEBUG:teuthology.orchestra.run.smithi143:> rm -rf -- /home/ubuntu/cephtest/archive 2024-06-15T00:59:58.289 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-06-15T00:59:58.300 INFO:teuthology.task.internal:Not uploading archives. 2024-06-15T00:59:58.300 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-06-15T00:59:58.322 INFO:teuthology.task.internal:Tidying up after the test... 2024-06-15T00:59:58.323 DEBUG:teuthology.orchestra.run.smithi105:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T00:59:58.326 DEBUG:teuthology.orchestra.run.smithi143:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-06-15T00:59:58.338 INFO:teuthology.orchestra.run.smithi105.stdout: 265304 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 15 00:59 /home/ubuntu/cephtest 2024-06-15T00:59:58.344 INFO:teuthology.orchestra.run.smithi143.stdout: 265305 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jun 15 00:59 /home/ubuntu/cephtest 2024-06-15T00:59:58.345 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-06-15T00:59:58.354 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-06-15T00:59:58.421 INFO:teuthology.run:Summary data: description: orch/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_9.stream conf/{client mds mon osd} fail_fs/no overrides/{ignorelist_health ignorelist_upgrade ignorelist_wrongly_marked_down pg-warn syntax} roles tasks/{0-from/quincy 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client/kclient 3-upgrade-mgr-staggered 4-config-upgrade/{fail_fs} 5-upgrade-with-workload 6-verify}} duration: 1194.2054960727692 failure_reason: timeout expired in wait_until_healthy flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=91ee14e767cb4b8fba5b92e6155f2632 status: fail success: false 2024-06-15T00:59:58.422 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-06-15T00:59:58.501 INFO:teuthology.run:FAIL